We all know how important a good backup is, but not everyone gives it the attention it deserves. Good backup characteristics include being current, consistent, durable, available, and secure.
Our object storage can perform some tasks - it is available worldwide, 24⁄7, it is durable and secure through the mechanism of transmission encryption, and through 3 copies of data stored on various nodes of the infrastructure. However, this is only half the success. It is still necessary to have a tool to send our backup to such a secure infrastructure. An interesting choice is the Duply project. As a Duplicity overlay, it enables you to easily and securely create backups of data and infrastructure. This tool works great with our object storage and has a number of useful features such as full encryption of stored data.
We will be using Ubuntu GNU/Linux version 18.04 for our work with Duply. We install Duply, Duplicity, and the Boto library:
sudo apt-get install duply duplicity python-boto
We create a new Duply configuration named e24backup:
sudo duply e24backup create
Now we will create keys for GnuPG with the gpg genkey command to encrypt backups:
sudo gpg --gen-key
gpg (GnuPG) 1.4.11; Copyright (C) 2010 Free Software Foundation, Inc. This is free software: you are free to change
and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want: (1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only) Your selection?
The default option is appropriate, confirm with the Enter key.
In the next step, we select the key size:
RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048)
The default 2048 bits is suitable for most purposes.
Please specify how long the key should be valid. 0 = key does not expire
= key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years
Key is valid for? (0)
Key does not expire at all Is this correct? (y/N)
In response to the question How long should the key be valid? (Key is valid for?), we can confirm the default of 0. Additionally, we agree to an unlimited key lifespan. We complete the identification information for the key:
You need a user ID to identify your key; the software constructs the user ID from the Real Name,
Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) "
Real name: e24backup
Email address: e24backup@foo.bar.com Comment: klucz
You selected this USERID:
"e24backup (klucz)"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O You need a Passphrase to protect your secret key.
We provide a password for the key (CAUTION: The password must be sufficiently strong):
You need a Passphrase to protect your secret key.
Enter Passphrase:
The system needs more time to gather enough entropy:
We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard,
move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better
chance to gain enough entropy.
When the system collects the appropriate amount of random data, it generates a key pair:
+++++
...+++++
We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard,
move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better
chance to gain enough entropy.
.........+++++
......+++++
gpg: key 07DE5301 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 1 signed: 0 trust: 0, 0q,
0n, 0m, 0f, 1u
pub 2048R/07DE5301 20130722
Key fingerprint = 510A 513E 64E6 2592 A94C 4742 32D6 218E 07DE 5301 uid e24backup
sub 2048R/8BCCE6BB 20130722
In this example, our key is 07DE5301. We edit the configuration file:
sudo vi /root/.duply/e24backup/conf
We find the lines:
GPG_KEY='_KEY_ID_'
GPG_PW='_GPG_PASSWORD_'
And then change them to our settings:
GPG_KEY='07DE5301'
GPG_PW='SUPER_long2_and_3random4pass5_xc%'
In the next step, we locate the next two lines:
TARGET='scheme://user[:password]@host[:port]/[/]path'
SOURCE='/path/of/source'
We change them to the API e24files settings that suit us, and the path that will be backed up SOURCE.
We recommend providing API keys in separate variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
The container in the example should have a unique name. We obtain it by adding a random suffix to our chosen name:
TARGET='s3://e24files.com/<bucket name>/<folder name>'
export AWS_ACCESS_KEY_ID='ACCESS_KEY'
export AWS_SECRET_ACCESS_KEY='SECRET_KEY'
SOURCE='/var/www'
Creating a backup of data looks like this:
sudo duply e24backup backup
Start duply v1.5.4.2, time is 20130722 12:08:15.
Using profile '/root/.duply/e24backup.
awk: warning: escape sequence `\/' treated as plain `/'
Using installed duplicity version 0.6.20, python 3.2.3, gpg 2.0.20 (Home: ~/.gnupg), awk 'GN Awk 4.0.0',
bash '4.2.37(1)release (x86_64pclinuxgnu)'.
Autoset first GPG_KEY entry 07DE5301 as signing key. Test Encrypt to 07DE5301 & Sign with 07DE5301 (OK)
Test Decrypt (OK)
Test Compare (OK)
Cleanup Delete '/tmp/duply.17622.1374487695_*'(OK)
Start running command PRE at 12:08:15.900
Skipping n/a script '/root/.duply/e24backup/pre'.
Finished state OK at 12:08:15.919 Runtime 00:00:00.018
Start running command BKP at 12:08:15.936
Import of duplicity.backends.u1backend Failed: No module named httplib2 Reading globbing filelist
/root/.duply/e24backup/exclude
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Jul 19 14:26:56 2013
Reuse configured PASSPHRASE as SIGN_PASSPHRASE
[ Backup Statistics ]
StartTime 1374487697.00 (Mon Jul 22 12:08:17 2013)
EndTime 1374488510.39 (Mon Jul 22 12:21:50 2013)
ElapsedTime 813.39 (13 minutes 33.39 seconds)
SourceFiles 584741
SourceFileSize 41643164600 (38.8 GB)
NewFiles 8805
NewFileSize 1064401783 (1015 MB)
DeletedFiles 4815
ChangedFiles 2125
ChangedFileSize 13861683876 (12.9 GB) ChangedDeltaSize 0 (0 bytes)
DeltaEntries 15745
RawDeltaSize 2792618932 (2.60 GB) TotalDestinationSizeChange 977175430 (932 MB) Errors 4
Finished state OK at 12:22:05.366 Runtime 00:13:49.430
Start running command POST at 12:22:05.385
Skipping n/a script '/root/.duply/e24backup/post'.
Finished state OK at 12:22:05.405 Runtime 00:00:00.020
We can check the status of our copy with the command:
sudo duply e24backup status
Start duply v1.5.4.2, time is 20130722 13:47:15.
Using profile '/root/.duply/e24backup'.
awk: warning: escape sequence `\/' treated as plain `/'
Using installed duplicity version 0.6.20, python 3.2.3, gpg 2.0.20 (Home: ~/.gnupg), awk 'GN Awk 4.0.0',
bash '4.2.37(1)release (x86_64pclinuxgnu)'.
Autoset first GPG_KEY entry 07DE5301 as signing key. Test Encrypt to 07DE5301 & Sign with 07DE5301 (OK)
Test Decrypt (OK)
Test Compare (OK)
Cleanup Delete '/tmp/duply.21319.1374493636_*'(OK)
Start running command STATUS at 13:47:16.496
Local and Remote metadata are synchronized, no sync needed. Last full backup date: Fri Jul 19 14:26:56 2013
Collection Status
Connecting with backend: BotoBackend
Archive dir: /root/.cache/e24backup/duply_e24backup
Found 0 secondary backup chains.
Found primary backup chain with matching signature chain:
Chain start time: Fri Jul 19 14:26:56 2013
Chain end time: Mon Jul 22 12:08:16 2013
Number of contained backup sets: 2
Total number of contained volumes: 471
Type of backup set: Time: Num volumes:
Full Fri Jul 19 14:26:56 2013 Incremental Mon Jul 22 12:08:16 2013
433
38
No orphaned or incomplete backup sets found.
Finished state OK at 13:47:17.295 Runtime 00:00:00.799
Listing files in the backup:
sudo duply e24backup list
Using profile '/root/.duply/e24backup.
Using installed duplicity version 0.6.20, python 3.2.3, gpg 2.0.20 (Home: ~/.gnupg), awk 'GN Awk 4.0.0',
bash '4.2.37(1)release (x86_64pclinuxgnu)'.
Autoset first GPG_KEY entry 07DE5301 as signing key.
Test Encrypt to 07DE5301 & Sign with 07DE5301 (OK)
Test Decrypt (OK)
Test Compare (OK)
Cleanup Delete '/tmp/duply.21569.1374493745_*'(OK)
Start running command LIST at 13:49:05.978
Local and Remote metadata are synchronized, no sync needed. Last full backup date: Fri Jul 19 14:26:56 2013
Fri Jul 19 09:40:32 2013 .
Thu Jul 4 14:01:48 2013 bin
Fri Jul 13 16:37:07 2012 bin/attr
Fri Jul 13 15:47:08 2012 bin/awk
Fri Jul 13 16:42:29 2012 bin/basename
Wed Feb 27 11:00:37 2013 bin/bash
Mon Jul 9 18:30:53 2012 bin/bashlogin
Fri Jul 13 17:44:25 2012 bin/bb
Fri Jul 13 15:04:02 2012 bin/bunzip2
Fri Jul 13 16:42:29 2012 bin/dir
Fri Jul 13 16:42:29 2012 bin/dirname
Fri Jul 13 15:44:11 2012 bin/dmesg
[..]
Recovering files from 4 days ago:
sudo duply e24backup restore /directory_to_write_recovered_data
Recovering one file from a week ago:
sudo duply e24backup fetch path_to_backup target_path
Clearing old backups:
sudo duply e24backup purge force sudo duply e24backup purgefull force
We can also configure Duply to use two keys to encrypt the backup. This way, we won’t store keys in text form so that in
the event of a security breach on the backup server, the backup cannot be decrypted. Duply defaults to incremental backups,
we can use configuration parameters such as MAX_AGE and MAX_FULLBACKUP_AGE to control the amount of stored data.