Backup Xen virtual machines with LVM snapshots and ftplicity/duplicity
Some time ago, I updated the backup system on a Server running multiple Xen VM instances (DomUs). Before changing the system, each virtual machine ran its own backup scripts to backup data to an external FTP server. Now, VMs are centrally backed up to FTP from the Dom0 using LVM (Logical Volume Manager) snapshots. As a backup solution I chose duplicity and ftplicity in combination with a shellscript to create automated LVM snapshots. Duplicity is a tool to create GPG-encrypted (this way you can store your backups at remote servers without having to worry about who has access to your data) incremental backups to remote servers, ftplicity is a wrapper script for duplicity which allows running duplicity without interaction (e.g. without the need to type any passwords). Ftplicity was originally published by the German computer magazine c’t, but has been undergone further development and is now hosted at SourceForge.
You can find tutorials on ftplicity/duplicity here (Note: they use the original c’t version of ftplicity):
- http://www.weareroot.de/2007/08/17/ftplicity-inkrementelle-backups-sicher-ablegen/ (German)
Basically you can use this setup for any kind of LVM snapshot based system, but I’m focusing on backing up Xen VMs here. I assume you got your LVM and Xen system up and running so far. I did this on a Debian Lenny system, but it should be similar on other distros. I did all steps as root.
Setup and Goal
My setup is the following: I got 1 volume group (VG) which contains all logical volumes (LV) used for virtual machines. Every virtual machine has 2 LVs, vm01-disk and vm01-swap, where vm01 is the name of the VM. Of course I want to backup only the *-disk LVs.
Goal: automatically create a LVM snapshot, mount it and back up the VM to the remote FTP server. Afterwards unmount the snapshot and remove it.
To automatically create LVM snapshots, I wrote a simple bash script which does all the needed steps. As my bash mojo isn’t too advanced, I got some ideas from here ;). You can download the script at Github.
Disclaimer: I do neither issue any guarantee that the script will work for you nor do I take any responsibility for potential data loss caused by this script. So please use with care.
The script has some options to configure:
- Path to your volume group (VG), e.g. /dev/lvmstore.
- Extension which will be added to LV names, e.g. if you specify vm01 as LV name and LVMEXTENSION is set to -disk, the snapshot will be created from /dev/lvmstore/vm01-disk (using the LVMPATH example from before).
- Path where snapshots will be mounted to.
- Size of the snapshot.
- An identifier, which will be used to create the snapshot name (useful to distinguish automatic backups from others). Using the examples from the other options, the snapshot will be named as follows: vm01-disk-snapshot-backupscript (with IDENTIFIER set to backupscript)
Edit the options according to your needs, make the script executable and drop it somewhere on your system. Example:
$ nano lvmsnapshot.sh [ ... edit options ...] $ chmod 700 lvmsnapshot.sh $ mv lvmsnapshot.sh /usr/sbin/lvmsnapshot
Assuming all options are set correctly for your system, you can use the script as follows. Create and mount a snapshot of virtual machine vm01, located at /dev/lvmstore/vm01-disk:
$ lvmsnapshot create vm01 Checking if /dev/lvmstore/vm01-disk is mounted...No Checking availability of Volume '/dev/lvmstore/vm01-disk'... ...successful Creating LVM snapshot at /dev/lvmstore/vm01-disk-snapshot-backupscript... Logical volume "vm01-disk-snapshot-backupscript" created ...successful Mounting LVM snapshot for backup... Creating mount directory at /mnt/lvm/vm01...OK ...successful
Umount and remove the snapshot:
$ lvmsnapshot remove vm01 Checking if /dev/lvmstore/vm01-disk is mounted...Yes Checking availability of Volume '/dev/lvmstore/vm01-disk'... ...successful Unmounting LVM snapshot after backup... Deleting mount directory at /mnt/lvm/vm01...OK ...successful Deleting LVM snapshot vm01-disk-snapshot-backupscript Logical volume "vm01-disk-snapshot-backupscript" successfully removed ...successful
If this works, we are able to automatically create snapshots and mount them. So we can pass to the next step: setting up ftplicity and configuring it to use the script to get access to our VM data.
First of all, install duplicity:
$ aptitude install duplicity ncftp
Get ftplicity from the SourceForge project site and make it executable:
$ wget http://downloads.sourceforge.net/sourceforge/ftplicity/ftplicity_1.4.2.tgz $ tar xvzf ftplicity_1.4.2.tgz $ cp ftplicity_1.4.2/ftplicity /usr/sbin/ftplicity $ chmod 700 /usr/sbin/ftplicity $ rm -rf ftplicity_1.4.2 ftplicity_1.4.2.tgz
Afterwards, create /etc/ftplicity. This way, ftplicity profiles will be stored in /etc instead of root‘s home directory.
Calling ftplicity should now work and tell you to specify a valid profile.
Create a GPG key
In order to be able to encrypt your backups, you have to create a GPG key. The tutorials mentioned at the beginning explain this in detail, so here’s only the short version. Open a second shell and run the following command (this generates some “randomness” on your system, which will be useful to create a secure key). Kill the command with CTRL+C when you are done with key generation.
while /bin/true; do cat /var/log/syslog > ~/temp.txt; sleep 1; done;
On your other shell, create your GPG key. Be sure to use a secure passphrase and to copy/write down the key ID which is displayed at the end of the generation process (we’ll need it for ftplicity). Also, make sure to backup the key to a secure location outside your server. As all your backups will be encrypted, they will be worthless if your server crashes and you lose the key.
Default options should be fine. This will create your key in ~/.gnupg/.
Set up ftplicity profiles
Now it’s time to set up a ftplicity profile for a virtual machine. The virtual machine is called vm01 and is running a webserver stack using Apache/MySQL. Call ftplicity to initialize the profile vm01:
$ ftplicity vm01 create Warning: The profile's folder '/etc/ftplicity/vm01' permissions were not safe (drwxr-xr-x). Secured them to 700. Congratulations. You just created the profile 'vm01'. The initial config file has been created as '/etc/ftplicity/vm01/conf'. For ftplicity to work you have to insert details on the gpg key to use and the ftp server for the backup in this config file. IMPORTANT: Copy the _whole_ profile folder after the first backup to a safe place. It contains everything needed to restore your backups. You will need it if you have to restore the backup from another system (e.g. after a system crash). Keep access to these files restricted as they contain _all_ informations (gpg data, ftp data) to access and modify your backups. Repeat this step after all configuration changes. Some configuration options are crucial for restoration.
Ftplicity creates a config file for your profile in /etc/ftplicity/vm01/conf, which you will have to edit to match your system. I changed the following options:
# gpg key data GPG_KEY='keyid' GPG_PW='keypassphrase' # ... # credentials & server address of the ftp server (URL-Format) TARGET='ftp://username@ftpserver/vm01' TARGET_PW='ftppassword' # base directory to backup SOURCE='/mnt/lvm/vm01' # ... # activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3) # forces a full backup if last full backup reaches a specified age, for the # format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS MAX_FULLBKP_AGE=6D DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
Make sure the directory on the FTP server exists. If you get any Python problems when running the backups, try to create an empty file in your backup directory on the FTP server (there is or was a Python bug which caused problems with empty directories when connecting to FTP).
Next, we have to tell ftplicity to use the snapshot-script to get access to vm01‘s data. Fortunately, ftplicity supports pre and post scripts, which are executed before and after a backup job. Create these scripts and make them executable:
$ cd /etc/ftplicity/vm01 $ touch pre $ touch post $ chmod 700 pre $ chmod 700 post
#!/bin/bash /usr/sbin/lvmsnapshot create vm01
#!/bin/bash /usr/sbin/lvmsnapshot remove vm01
As a last step, create a file /etc/ftplicity/vm01/exclude which tells ftplicity which files to exclude from the backup. Note that I excluded /var/lib/mysql too, as backing up the MySQL data directory could lead to inconsistent results. To solve this, I use automysqlbackup inside VMs which dumps SQL data regularly. These dumps are included in my FTP backup. I’ll write more on automysqlbackup in a later tutorial.
/mnt/lvm/vm01/var/run/** /mnt/lvm/vm01/var/tmp/** /mnt/lvm/vm01/tmp/** /mnt/lvm/vm01/dev/** /mnt/lvm/vm01/sys/** /mnt/lvm/vm01/proc/** /mnt/lvm/vm01/floppy/** /mnt/lvm/vm01/cdrom/** /mnt/lvm/vm01/var/lib/mysql/**
OK, all is set up and should be working now. So let’s try using ftplicity. You can see all ftplicity commands by calling ftplicity usage.
Check status of our profile:
$ ftplicity vm01 status Start ftplicity v1.4.2, time is 07/08/09 20:09:47. Using profile '/etc/ftplicity/vm01'. Using installed duplicity version 0.4.11, gpg 1.4.9 (Home: ~/.gnupg) Test - Encryption with key xxxxxxxx (OK) Test - Decryption with key xxxxxxxx (OK) Test - Compare Original w/ Decryption (OK) Cleanup - Delete '/tmp/ftplicity.xxxxx.xxxxxxxxxx_*'(OK) --- Start running command STATUS (20:09:47.943) --- Running duplicity - OK Output: NcFTP version is 3.2.1 Last full backup date: none Connecting with backend: ftpBackend Archive dir: None Found 0 backup chains without signatures. No backup chains with active signatures found No orphaned or incomplete backup sets found. --- Finished (20:09:48.206) - Runtime 00:00:00.263 ---
OK, profile seems to work. Let’s try to make a backup.
ftplicity vm01 backup Start ftplicity v1.4.2, time is 07/08/09 20:14:57. Using profile '/etc/ftplicity/vm01'. Using installed duplicity version 0.4.11, gpg 1.4.9 (Home: ~/.gnupg) Test - Encryption with key xxxxxxxx (OK) Test - Decryption with key xxxxxxxx (OK) Test - Compare Original w/ Decryption (OK) Cleanup - Delete '/tmp/ftplicity.xxxxx.xxxxxxxxxx_*'(OK) --- Start running command PRE (20:14:58.152) --- Running '/etc/ftplicity/vm01/pre' - OK Output: Checking if /dev/lvmstore/vm01-disk is mounted...No Checking availability of Volume '/dev/lvmstore/vm01-disk'... ...successful Creating LVM snapshot at /dev/lvmstore/vm01-disk-snapshot-backupscript... Logical volume "vm01-disk-snapshot-backupscript" created ...successful Mounting LVM snapshot for backup... Creating mount directory at /mnt/lvm/vm01...OK ...successful --- Finished (20:14:58.490) - Runtime 00:00:00.338 --- --- Start running command BKP (20:14:58.498) --- Running duplicity - OK Output: NcFTP version is 3.2.1 Reading globbing filelist /etc/ftplicity/vm01/exclude Last full backup date: none Last full backup is too old, forcing full backup --------------[ Backup Statistics ]-------------- StartTime 1247076898.77 (Wed Jul 8 20:14:58 2009) EndTime 1247076977.92 (Wed Jul 8 20:16:17 2009) ElapsedTime 79.15 (1 minute 19.15 seconds) SourceFiles 12398 SourceFileSize 276924030 (264 MB) NewFiles 12398 NewFileSize 276924030 (264 MB) DeletedFiles 0 ChangedFiles 0 ChangedFileSize 0 (0 bytes) ChangedDeltaSize 0 (0 bytes) DeltaEntries 12398 RawDeltaSize 130935522 (125 MB) TotalDestinationSizeChange 101194951 (96.5 MB) Errors 0 ------------------------------------------------- --- Finished (20:16:19.864) - Runtime 00:01:21.365 --- --- Start running command POST (20:16:19.872) --- Running '/etc/ftplicity/vm01/post' - OK Output: Checking if /dev/lvmstore/vm01-disk is mounted...Yes Checking availability of Volume '/dev/lvmstore/vm01-disk'... ...successful Unmounting LVM snapshot after backup... Deleting mount directory at /mnt/lvm/vm01...OK ...successful Deleting LVM snapshot vm01-disk-snapshot-backupscript Logical volume "vm01-disk-snapshot-backupscript" successfully removed ...successful --- Finished (20:16:20.718) - Runtime 00:00:00.846 ---
Perfect. The LVM snapshot is created and mounted, backuped and afterwards all gets cleaned up. As this was the first backup, ftplicity created a full backup. Following backups will be incremental until the full backup is too old (see ftplicity config for details).
Ftplicity allows simple restoring of your backups. Some examples:
Restore the complete last state to /tmp/vm01restore:
$ ftplicity vm01 restore /tmp/vm01restore
Restore /etc/passwd from the last backup state to /tmp/vm01restore/etc/passwd:
$ ftplicity vm01 fetch etc/passwd /tmp/vm01restore/etc/passwd
The same as before, but take the state from four days before:
$ ftplicity vm01 fetch etc/passwd /tmp/vm01restore/etc/passwd 4D
Set up cronjobs
To run a backup automatically every night, I set up cronjobs to do this. Example:
0 1 * * * /usr/sbin/ftplicity vm01 cleanup --force ; /usr/sbin/ftplicity vm01 backup 0 2 * * 1 /usr/sbin/ftplicity vm01 purge-full --force ; /usr/sbin/ftplicity vm01 purge --force
This will run ftplicity on vm01 every night at 1:00 and purge old backups every monday night at 2:00.
The above setup gives you a secure and handy solution to run automated backups on LVM snapshots. Ftplicity is quite simple to use and gives you many possibilities to restore your files. To add more VMs, just create a new profile (or copy the existing one) and adjust the config files to match the new VM paths. Any suggestions are welcome :)