Backup Xen virtual machines with LVM snapshots and ftplicity/duplicity

July 8, 2009 · in How-Tos, Linux, Server

Some time ago, I updated the backup system on a Server running multiple Xen VM instances (DomUs). Before changing the system, each virtual machine ran its own backup scripts to backup data to an external FTP server. Now, VMs are centrally backed up to FTP from the Dom0 using LVM (Logical Volume Manager) snapshots. As a backup solution I chose duplicity and ftplicity in combination with a shellscript to create automated LVM snapshots. Duplicity is a tool to create GPG-encrypted (this way you can store your backups at remote servers without having to worry about who has access to your data) incremental backups to remote servers, ftplicity is a wrapper script for duplicity which allows running duplicity without interaction (e.g. without the need to type any passwords). Ftplicity was originally published by the German computer magazine c’t, but has been undergone further development and is now hosted at SourceForge.

You can find tutorials on ftplicity/duplicity here (Note: they use the original c’t version of ftplicity):

Basically you can use this setup for any kind of LVM snapshot based system, but I’m focusing on backing up Xen VMs here. I assume you got your LVM and Xen system up and running so far. I did this on a Debian Lenny system, but it should be similar on other distros. I did all steps as root.

Setup and Goal

My setup is the following: I got 1 volume group (VG) which contains all logical volumes (LV) used for virtual machines. Every virtual machine has 2 LVs, vm01-disk and vm01-swap, where vm01 is the name of the VM. Of course I want to backup only the *-disk LVs.

Goal: automatically create a LVM snapshot, mount it and back up the VM to the remote FTP server. Afterwards unmount the snapshot and remove it.

LVM snapshots

To automatically create LVM snapshots, I wrote a simple bash script which does all the needed steps. As my bash mojo isn’t too advanced, I got some ideas from here ;). You can download the script at Github.

Disclaimer: I do neither issue any guarantee that the script will work for you nor do I take any responsibility for potential data loss caused by this script. So please use with care.

The script has some options to configure:

LVMPATH: Path to your volume group (VG), e.g. /dev/lvmstore.
LVMEXTENSION: Extension which will be added to LV names, e.g. if you specify vm01 as LV name and LVMEXTENSION is set to -disk, the snapshot will be created from /dev/lvmstore/vm01-disk (using the LVMPATH example from before).
MOUNTPATH: Path where snapshots will be mounted to.
SNAPSHOTSIZE: Size of the snapshot.
IDENTIFIER: An identifier, which will be used to create the snapshot name (useful to distinguish automatic backups from others). Using the examples from the other options, the snapshot will be named as follows: vm01-disk-snapshot-backupscript (with IDENTIFIER set to backupscript)

Edit the options according to your needs, make the script executable and drop it somewhere on your system. Example:

$ nano lvmsnapshot.sh
[ ... edit options ...]
$ chmod 700 lvmsnapshot.sh
$ mv lvmsnapshot.sh /usr/sbin/lvmsnapshot

Assuming all options are set correctly for your system, you can use the script as follows. Create and mount a snapshot of virtual machine vm01, located at /dev/lvmstore/vm01-disk:

$ lvmsnapshot create vm01
Checking if /dev/lvmstore/vm01-disk is mounted...No
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Creating LVM snapshot at /dev/lvmstore/vm01-disk-snapshot-backupscript...
  Logical volume "vm01-disk-snapshot-backupscript" created
...successful

Mounting LVM snapshot for backup...
  Creating mount directory at /mnt/lvm/vm01...OK
...successful

Umount and remove the snapshot:

$ lvmsnapshot remove vm01
Checking if /dev/lvmstore/vm01-disk is mounted...Yes
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Unmounting LVM snapshot after backup...
  Deleting mount directory at /mnt/lvm/vm01...OK
...successful

Deleting LVM snapshot vm01-disk-snapshot-backupscript
  Logical volume "vm01-disk-snapshot-backupscript" successfully removed
...successful

If this works, we are able to automatically create snapshots and mount them. So we can pass to the next step: setting up ftplicity and configuring it to use the script to get access to our VM data.

Install ftplicity/duplicity

First of all, install duplicity:

$ aptitude install duplicity ncftp

Get ftplicity from the SourceForge project site and make it executable:

$ wget http://downloads.sourceforge.net/sourceforge/ftplicity/ftplicity_1.4.2.tgz
$ tar xvzf ftplicity_1.4.2.tgz
$ cp ftplicity_1.4.2/ftplicity /usr/sbin/ftplicity
$ chmod 700 /usr/sbin/ftplicity
$ rm -rf ftplicity_1.4.2 ftplicity_1.4.2.tgz

Afterwards, create /etc/ftplicity. This way, ftplicity profiles will be stored in /etc instead of root‘s home directory.

mkdir /etc/ftplicity

Calling ftplicity should now work and tell you to specify a valid profile.

Create a GPG key

In order to be able to encrypt your backups, you have to create a GPG key. The tutorials mentioned at the beginning explain this in detail, so here’s only the short version. Open a second shell and run the following command (this generates some “randomness” on your system, which will be useful to create a secure key). Kill the command with CTRL+C when you are done with key generation.

while /bin/true; do cat /var/log/syslog > ~/temp.txt; sleep 1; done;

On your other shell, create your GPG key. Be sure to use a secure passphrase and to copy/write down the key ID which is displayed at the end of the generation process (we’ll need it for ftplicity). Also, make sure to backup the key to a secure location outside your server. As all your backups will be encrypted, they will be worthless if your server crashes and you lose the key.

gpg --gen-key

Default options should be fine. This will create your key in ~/.gnupg/.

Set up ftplicity profiles

Now it’s time to set up a ftplicity profile for a virtual machine. The virtual machine is called vm01 and is running a webserver stack using Apache/MySQL. Call ftplicity to initialize the profile vm01:

$ ftplicity vm01 create

Warning:

The profile's folder
'/etc/ftplicity/vm01'
permissions were not safe (drwxr-xr-x). Secured them to 700.

Congratulations. You just created the profile 'vm01'.
The initial config file has been created as
'/etc/ftplicity/vm01/conf'.
For ftplicity to work you have to insert details on
the gpg key to use and the ftp server for the backup
in this config file.

IMPORTANT:
  Copy the _whole_ profile folder after the first backup to a safe place.
  It contains everything needed to restore your backups. You will need
  it if you have to restore the backup from another system (e.g. after a
  system crash). Keep access to these files restricted as they contain
  _all_ informations (gpg data, ftp data) to access and modify your backups.

  Repeat this step after all configuration changes. Some configuration
  options are crucial for restoration.

Ftplicity creates a config file for your profile in /etc/ftplicity/vm01/conf, which you will have to edit to match your system. I changed the following options:

# gpg key data
GPG_KEY='keyid'
GPG_PW='keypassphrase'

# ...

# credentials & server address of the ftp server (URL-Format)
TARGET='ftp://username@ftpserver/vm01'
TARGET_PW='ftppassword'

# base directory to backup
SOURCE='/mnt/lvm/vm01'

# ...

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
MAX_FULLBKP_AGE=6D
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

Make sure the directory on the FTP server exists. If you get any Python problems when running the backups, try to create an empty file in your backup directory on the FTP server (there is or was a Python bug which caused problems with empty directories when connecting to FTP).

Next, we have to tell ftplicity to use the snapshot-script to get access to vm01‘s data. Fortunately, ftplicity supports pre and post scripts, which are executed before and after a backup job. Create these scripts and make them executable:

$ cd /etc/ftplicity/vm01
$ touch pre
$ touch post
$ chmod 700 pre
$ chmod 700 post

pre

#!/bin/bash
/usr/sbin/lvmsnapshot create vm01

post

#!/bin/bash
/usr/sbin/lvmsnapshot remove vm01

As a last step, create a file /etc/ftplicity/vm01/exclude which tells ftplicity which files to exclude from the backup. Note that I excluded /var/lib/mysql too, as backing up the MySQL data directory could lead to inconsistent results. To solve this, I use automysqlbackup inside VMs which dumps SQL data regularly. These dumps are included in my FTP backup. I’ll write more on automysqlbackup in a later tutorial.

/mnt/lvm/vm01/var/run/**
/mnt/lvm/vm01/var/tmp/**
/mnt/lvm/vm01/tmp/**
/mnt/lvm/vm01/dev/**
/mnt/lvm/vm01/sys/**
/mnt/lvm/vm01/proc/**
/mnt/lvm/vm01/floppy/**
/mnt/lvm/vm01/cdrom/**
/mnt/lvm/vm01/var/lib/mysql/**

Run ftplicity

OK, all is set up and should be working now. So let’s try using ftplicity. You can see all ftplicity commands by calling ftplicity usage.

Check status of our profile:

$ ftplicity vm01 status
Start ftplicity v1.4.2, time is 07/08/09 20:09:47.
Using profile '/etc/ftplicity/vm01'.
Using installed duplicity version 0.4.11, gpg 1.4.9 (Home: ~/.gnupg)
Test - Encryption with key xxxxxxxx (OK)
Test - Decryption with key xxxxxxxx (OK)
Test - Compare Original w/ Decryption (OK)
Cleanup - Delete '/tmp/ftplicity.xxxxx.xxxxxxxxxx_*'(OK)

--- Start running command STATUS (20:09:47.943) ---
Running duplicity - OK
Output: NcFTP version is 3.2.1
Last full backup date: none
Connecting with backend: ftpBackend
Archive dir: None

Found 0 backup chains without signatures.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
--- Finished (20:09:48.206) - Runtime 00:00:00.263 ---

OK, profile seems to work. Let’s try to make a backup.

ftplicity vm01 backup
Start ftplicity v1.4.2, time is 07/08/09 20:14:57.
Using profile '/etc/ftplicity/vm01'.
Using installed duplicity version 0.4.11, gpg 1.4.9 (Home: ~/.gnupg)
Test - Encryption with key xxxxxxxx (OK)
Test - Decryption with key xxxxxxxx (OK)
Test - Compare Original w/ Decryption (OK)
Cleanup - Delete '/tmp/ftplicity.xxxxx.xxxxxxxxxx_*'(OK)

--- Start running command PRE (20:14:58.152) ---
Running '/etc/ftplicity/vm01/pre' - OK
Output: Checking if /dev/lvmstore/vm01-disk is mounted...No
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Creating LVM snapshot at /dev/lvmstore/vm01-disk-snapshot-backupscript...
  Logical volume "vm01-disk-snapshot-backupscript" created
...successful

Mounting LVM snapshot for backup...
  Creating mount directory at /mnt/lvm/vm01...OK
...successful
--- Finished (20:14:58.490) - Runtime 00:00:00.338 ---

--- Start running command BKP (20:14:58.498) ---
Running duplicity - OK
Output: NcFTP version is 3.2.1
Reading globbing filelist /etc/ftplicity/vm01/exclude
Last full backup date: none
Last full backup is too old, forcing full backup
--------------[ Backup Statistics ]--------------
StartTime 1247076898.77 (Wed Jul  8 20:14:58 2009)
EndTime 1247076977.92 (Wed Jul  8 20:16:17 2009)
ElapsedTime 79.15 (1 minute 19.15 seconds)
SourceFiles 12398
SourceFileSize 276924030 (264 MB)
NewFiles 12398
NewFileSize 276924030 (264 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 12398
RawDeltaSize 130935522 (125 MB)
TotalDestinationSizeChange 101194951 (96.5 MB)
Errors 0
-------------------------------------------------
--- Finished (20:16:19.864) - Runtime 00:01:21.365 ---

--- Start running command POST (20:16:19.872) ---
Running '/etc/ftplicity/vm01/post' - OK
Output: Checking if /dev/lvmstore/vm01-disk is mounted...Yes
Checking availability of Volume '/dev/lvmstore/vm01-disk'...
  ...successful

Unmounting LVM snapshot after backup...
  Deleting mount directory at /mnt/lvm/vm01...OK
...successful

Deleting LVM snapshot vm01-disk-snapshot-backupscript
  Logical volume "vm01-disk-snapshot-backupscript" successfully removed
...successful
--- Finished (20:16:20.718) - Runtime 00:00:00.846 ---

Perfect. The LVM snapshot is created and mounted, backuped and afterwards all gets cleaned up. As this was the first backup, ftplicity created a full backup. Following backups will be incremental until the full backup is too old (see ftplicity config for details).

Restore

Ftplicity allows simple restoring of your backups. Some examples:

Restore the complete last state to /tmp/vm01restore:

$ ftplicity vm01 restore /tmp/vm01restore

Restore /etc/passwd from the last backup state to /tmp/vm01restore/etc/passwd:

$ ftplicity vm01 fetch etc/passwd /tmp/vm01restore/etc/passwd

The same as before, but take the state from four days before:

$ ftplicity vm01 fetch etc/passwd /tmp/vm01restore/etc/passwd 4D

Perfect.

Set up cronjobs

To run a backup automatically every night, I set up cronjobs to do this. Example:

0 1 * * * /usr/sbin/ftplicity vm01 cleanup --force ; /usr/sbin/ftplicity vm01 backup
0 2 * * 1 /usr/sbin/ftplicity vm01 purge-full --force ; /usr/sbin/ftplicity vm01 purge --force

This will run ftplicity on vm01 every night at 1:00 and purge old backups every monday night at 2:00.

Conlusion

The above setup gives you a secure and handy solution to run automated backups on LVM snapshots. Ftplicity is quite simple to use and gives you many possibilities to restore your files. To add more VMs, just create a new profile (or copy the existing one) and adjust the config files to match the new VM paths. Any suggestions are welcome :)

Tags: administration, duplicity, encryption, ftp, ftplicity, gpg, lvm, security, snapshot, vm, xen

http://www.orloseweight.com how to lose weight fast

This virtual appliance is equipped to configure with popular virtual environments such as VMWare, Microsoft’s Hyper-V, and XEN hypervisors and virtual …
http://the-movie-vault.com/watch-hall-pass-online/ watch hall pass online

You can then use tools provided with the hypervisor to snapshot the virtual machine in the background and move the copy via a dedicated backup server for …
http://365cigars.com cigar reviews

Dropbox is a backup and storage service that provides 2GB of space free for … the form of virtual machines on Amazon’s Xen-based virtualization platform.
http://www.adhost.dk/sogemaskineoptimering.shtml søgemaskineoptimering

While this method does reduce the number of agents that must be installed and maintained, it still requires that the administrator either shut down the virtual machines to back them up or back up the VMs as open files, creating “crash consistent” …
- http://www.facebook.com/original.matthieu.larcher Matthieu Larcher
  
  Would you have any better solution to suggest ? I liked what I saw in this post very much until I read about “crash consistent” backups :/
  - http://ailoo.net Mathias
    
    Well, backups are consistent, as LVM snapshots give you an exact state of the block device at creation time. You just have to handle some exceptional cases/applications like databases manually (that’s why I excluded the MySQL data directory) in the example config.
    
    Taking the MySQL example, you can still include them in your backup by dumping them to disk and backing up the dumps (using automysqlbackup for example).
    - http://www.facebook.com/original.matthieu.larcher Matthieu Larcher
      
      After looking at the various options, I eventually implemented what you described. Thanks for this very nice and clear tutorial by the way.
      I used percona’s xtrabackup instead of automysqlbackup. It seems to be the best hot backup tool there is now, it handles all engines, and it’s free and open source. What else could you ask for? ;) I’d recommend you check on it at http://www.percona.com/software/percona-xtrabackup/ There’s a good tutorial at http://marcus.bointon.com/archives/87-MySQL-backups-with-Perconas-XtraBackup.html
      Also, I had troubles creating the pgp key, as my box wasn’t generating enough entropy. I eventually solved it thanks to this post : https://www.wzdftpd.net/blog/index.php?post/2008/06/16/24-linux-kernel-and-entropy
http://www.eusuperpharmacy.com/ Kamagra next day delivery

Given typical VMware consolidation ratios in the range of six to 10 virtual machines (VMs) per host, a virtual server administrator must choose a backup …