Cumulus Linux Snapshot Rollback

After my first post about Cumulus Linux non-disruptive upgrade procedure on MLAG pairs here the rollback procedure which is good to know in case you need to revert a snapshot after an unsuccessful software upgrade or bigger configuration change which went wrong.

To back out from an upgrade, you can roll back the state of your switch (software and configuration) to an earlier snapshot.

The rollback is for sure disruptive!

The Rollback will revert the entire system except for logs and home directories. So any configuration changes made after the upgrade will also be reverted and lost.

To create a custom snapshot use the following command, otherwise during the software upgrade an snapshot is created automatically.

[email protected]:~$ sudo snapper create -d SNAPSHOT_NAME

To rollback a snapshot perform the following steps on each switch. View the list of snapshots on the switch using the following command

sudo snapper list

You will see output similar to the following:

[email protected]:~$ sudo snapper list
Type   | #  | Pre # | Date                            | User | Cleanup | Description                            | Userdata    
-------+----+-------+---------------------------------+------+---------+----------------------------------------+--------------
single | 0  |       |                                 | root |         | current                                |             
single | 1  |       | Sat 24 Sep 2016 01:45:36 AM UTC | root |         | first root filesystem                  |             
pre    | 20 |       | Thu 01 Dec 2016 01:43:29 AM UTC | root | number  | nclu pre  'net commit' (user cumulus)  |             
post   | 21 | 20    | Thu 01 Dec 2016 01:43:31 AM UTC | root | number  | nclu post 'net commit' (user cumulus)  |             
pre    | 22 |       | Thu 01 Dec 2016 01:44:18 AM UTC | root | number  | nclu pre  '20 rollback' (user cumulus) |             
post   | 23 | 22    | Thu 01 Dec 2016 01:44:18 AM UTC | root | number  | nclu post '20 rollback' (user cumulus) |             
single | 26 |       | Thu 01 Dec 2016 11:23:06 PM UTC | root |         | test_snapshot                          |             
pre    | 29 |       | Thu 01 Dec 2016 11:55:16 PM UTC | root | number  | pre-apt                                | important=yes
post   | 30 | 29    | Thu 01 Dec 2016 11:55:21 PM UTC | root | number  | post-apt                               | important=yes

You want to locate the “pre-apt” snapshot that corresponds to the date and time of the upgrade.  Once you have identified the snapshot, note the number from the second column of the table (#).

Determine the number on each switch and don’t assume the number is the same on both switches if you need to rollback.

For doing a rollback use the following command

sudo snapper rollback NUMBER#

You will see the following output:

[email protected]:~$ sudo snapper rollback 29
Creating read-only snapshot of current system. (Snapshot 31.)
Creating read-write snapshot of snapshot 29. (Snapshot 32.)
Setting default subvolume to snapshot 32.
[email protected]:~$

Note the snapshot number 31. reported for the “read-only snapshot of the current system”.   You can use this to revert the rollback if needed.

Reboot the system with the command

sudo reboot

More information you can find in the Cumulus Linux documentation:

https://docs.cumulusnetworks.com/display/DOCS/Using+Snapshots

F5 BIGIP Software upgrade procedure

Here is a procedure on how to upgrade the software on an F5 BIGIP cluster tier.

  1. Download last ISO version for F5 BIGIP LTM
  2. Upload the ISO to the standby unit
  3. Install the ISO to a free partition on the standby unit
  4. Change boot location to the new partition and device automatically reboots.

After 6 to 7 minutes the device booted with the new software version. From now on you won’t be able to sync your configs between the devices anymore because of the different software versions it’s only possible within the same release. F5 doesn’t recommend to run cluster with different software versions.

Now initiate the failover on the active unit! Do that maybe at low traffic period! Normally there should be no side effect and the failover is not visible to clients because of the mirroring the sessions.

In case of problems with the new software version you should be able to switch back to the other cluster member with the old software version at any time.

  1. When you verify that everything is working after the failover you can start the other device now
  2. Upload ISO file
  3. Install ISO file
  4. Change boot location

The same procedure you use to install hotfixes on the BigIP.