# Upgrade GitLab Version (Zero Downtime)

These are the steps to perform a zero downtime upgrade for minor releases.

# Preparation Steps

  1. Ensure that a calendar event has been created on the Demo Systems calendar for the maintenance window.

    US - GitLab v12.x Upgrade (Zero Downtime)

  2. Post in the #demo-systems Slack channel to remind users that the maintenance window is about to start.

    @here :alarm_clock: SCHEDULED UPGRADE :tanuki: Our maintenance window for upgrading to GitLab v12.x begins in 15 minutes (2:00pm-4:00pm PT, 21:00-23:00 UTC). This is a minor version upgrade from v12.x to v12.y. This will be a zero downtime upgrade, however there is a risk that services may be temporarily unavailable or restarting for https://gitlab-core.us.gitlabdemo.cloud.

  3. Log in to the GitLab instance https://gitlab-core.{REGION}.gitlabdemo.cloud with the root account.

  4. Navigate to Admin > Messages.

  5. Create or update the broadcast message with scheduled maintenance window time.

    • Message: We are currently performing zero downtime maintenance to upgrade from GitLab v12.x to v12.y. Please post issues in the #demo-systems channel.
    • Type: Banner
    • Background color: #E75E40
    • Dismissable: Checked/Enabled
    • Starts at (UTC): {Start Time}
    • Ends at (UTC): {End Time}

# Back Up the GitLab Instance

  1. Open a Terminal window with 2 tabs. You'll use one tab/window for connecting to the GitLab instance and another for performing local actions on your machine.
# Terminal Tab/Window 1
# Connect to the GitLab instance using SSH
ssh demosys@gitlab-core.{REGION}.gitlabdemo.cloud

# Elevate to root privileges
sudo -i

# Use the GitLab helper to start the backup process
# Note: This is built-in and can run from any directory
gitlab-backup create

# Note: The backup can take a long time (up to ~30 minutes)

# Navigate to the backups directory, modify the permissions and move to /tmp
# directory so that we can easily access it from our local machine
# TODO - We will move this to an S3/GCS bucket soon.
cd /var/opt/gitlab/backups/
ls -la

# Locate the new file based on timestamp
cp {FILENAME} /tmp
cd /tmp
chmod 755 {FILENAME}

# Leave this terminal window open
# Terminal Tab/Window 2
# Change to a directory that you want the backup file to be located
cd ~/Desktop

# Use SCP to transfer the backup file from the server to your machine
# This ensures that we have a backup just in-case we need to rebuild the instance
scp demosys@gitlab-core.us.gitlabdemo.cloud:/tmp/{filename} .

# Note: The transfer can take a long time (20-30 Minutes, ~50GB sometimes)
# Terminal Tab/Window 1
# Using your previous terminal window for the GitLab server

# Remove the backup file (since it takes up disk space)
rm /tmp/{FILENAME}

# Use Ansible Playbook to Perform Zero Downtime Upgrade

We have created an Ansible playbook to perform the upgrade using the steps documented in GitLab's product documentation for a zero-downtime upgrade. You can learn more about the Ansible playbook in the tasks file for the Ansible role.

  1. Open a new terminal tab or window.

  2. Navigate to the demosys-ansible repository on your local machine. If you have not cloned this yet, please see the README instructions on the demosys-ansible project.

    cd ~/Sites/gitlab-com/customer-success/demo-systems/infrastructure/

  3. Ensure that you're on the master branch. You may need to use your preferred Git client to stash your changes on your working branch before switching to master.

    git checkout master

  4. Perform a git pull to ensure that you have the latest code.

    git pull

  5. Run the following command to use the Ansible playbook to upgrade to the latest version. There are no variables to change except for the {REGION} (ex. us, eu, asia) in the command arguments.

    ansible-playbook task-upgrade-gitlab-version.yml --vault-password-file=keys/demosys-vault.pass --inventory=hosts/demosys-saas/{REGION}/hosts

  6. Once the playbook has been completed, the GitLab upgrade should be complete.

  7. Log in to the GitLab instance https://gitlab-core.{REGION}.gitlabdemo.cloud with the root account.

  8. Navigate to Admin > Overview > Dashboard.

  9. On the dashboard, check to see if the latest version number is shown in the Components card.

# Cleanup Steps

  1. Navigate to Admin > Messages.

  2. Remove the broadcast message.

  3. Post in the #demo-systems Slack channel to let users know that the maintenance window has been completed.

    @here :green_stop_light: Maintenance Window Completed. We are now running GitLab v12.x. Please post a new message (not a comment on this thread) if you have any questions or issues.