Simple Encrypted Incremental Backups

Simple encrypted incremental backups using a Raspberry Pi and USB disk

Posted by J├╝rgen on 8 october 2022

Creating encrypted incremental backups using a Pi and an external USB disk

Ransomware is getting more and more common these days. I gathered I should be creating a low-budget way to mitigate such a danger. I've been using a Raspberry Pi 1B with an external 3TB USB with rsync to at least keep my data in 2 places. But since this is a simple replication, when my data gets hijacked and encrypted, this gets replicated as well. So I decided to create a solution for creating backups with the possibility to go back in time.

Requirements

  • Encrypted backups (for security)
  • Non-interactive booting (when the setup reboots, it should be ready to continue without needing any interaction)
  • Secure encryption key (obtaining the encryption key should not be possible when one only has the encrypted disk + pi)
  • Snapshot schedule for different intervals
  • Incremental snapshots (When you have a big data set, it's not viable to create snapshots containing the complete dataset)
  • "Isolated" (no-one should be able to access the backup device when the network is compromised)
  • Notifications (Since it's isolated, it would be nice to know whether or not backups are succeeding)

What we have

  • Raspberry Pi 3 with debian-based linux distribution
  • Wired network connection
  • External 3TB USB disk

Solution

Encryption will be achieved by luksFormatting the external disk using a password. In order to boot the Pi Non-Interactively, the external encrypted disk needs to be mounted when needed without asking for a password. But to have the password secured, it must not exist on the Pi, in case the disk and Pi gets into wrong hands. So the plan is to have the password stored in a file (with sufficiently secure file permissions and encrypted using GPG) on the source storage and have the Pi read that file during the boot sequence. The GPG encryption is meant to have a crude 2-factor auth, meaning that the only way to get the secret from the secret.txt is to have both machines. The source machine has the secret.txt, but no key stored to decrypt it, the destination machine needs to have access to the source machine in order to access the secret.txt and to decrypt it using the key only it has.

In order to create snapshots, we'll be using rsnapshot. This software works based on hard-links in order to keep the bulk of the data de-duplicated, this hard-linking requires a filesystem which supports hard-links, like ext4. Moving hard-links around eliminates the need for periodic full backups. This is why I choose rsnapshot in favor of other software like duplicity.

The snapshot schedule will be defined in a combination of rsnapshot.conf and crontab.

It's not really possible to have the backups isolated (as in air-gapped) from the network. But as the Pi is the one mounting the source, and there are no open services/ports (including SSH), the isolation suffices. And because of that isolation, it is desirable to have the system sending out mails on successful backups, because you have no other way to check. Hourly notifications will be overkill, but daily or even weekly will be good enough.

Installation

First of all, make sure the Pi is installed with Raspberry Pi OS.

Install software

Install the following packages:


apt install -y cryptsetup rsnapshot screen sendemail gnupg

cryptsetup is the tool used to encrypt block devices. rsnapshot is the tool used to create snapshots. sendemail is used to be able to send out emails without the need to have an MTA installed. gnupg can be used to encrypt text, either symmetrical of asymmetrical. And lastly, with screen one can detach a shell, this is useful for creating the initial backup, which can take a considerable amount of time.

Create encrypted USB disk

Connect your USB disk, and luksformat the device:


luksformat -t ext4 /dev/sda

luksformat will query for a key (LUKS KEY) to encrypt your LUKS device, carefully keep this key, without it, you cannot access your device. Note; You could also used /dev/sd? as a wildcard, but be sure you have only one such block-device. -t ext4 is required, since rsnapshot is built around the use of hard-links, which the ext? filesystem supports.

Create decryption key

On your source medium, create an encrypted text-file containing the decryption key for your luks device.


echo "<LUKS KEY>" | gpg -c --batch --yes --passphrase <SECRET> - > secret.txt

Instead of storing your LUKS KEY in plain text on your source device, the text-file will be encrypted using gpg. The SECRET for decrypting this file will not be available on the source volume, but only to the Pi creating the backups.

Backup script

On your Pi place the following script as /root/backup.sh:


#!/usr/bin/bash

set -e

export PATH=/bin:/sbin:/usr/bin:/usr/sbin

if [ "$1" != "hourly" ] && [ "$1" != "daily" ] && [ "$1" != "weekly" ] && [ "$1" != "monthly" ]; then
    echo "provide any of [hourly|daily|weekly|monthly]."
    exit 1
fi

SOURCE=192.168.100.10:/medium/storage/
DESTINATION=/dev/sd?  # disk letter may change, but since it's the only disk, just wildcard it.
PASSWORD=********

# Get into the expected state, unmount storage and lock disk
umount /mnt/src | true
umount /mnt/dst | true
cryptsetup luksClose /dev/mapper/backup | true

# Mount the source medium, from there use an encrypted secret to unlock the external disk, and mount the unlocked volume.
mount $SOURCE /mnt/src
echo "$(cat /mnt/src/secret.txt)" | gpg -d --batch --yes --passphrase $PASSWORD | cryptsetup luksOpen $DESTINATION backup -
mount /dev/mapper/backup /mnt/dst

# Create the requested snapshot
rsnapshot $1
result=$?

if [ "$1" == "daily" ] && [ "$result" == "0" ]; then
        # In case this is a daily snapshot, send out an e-mail to notify that the backup was created,
        # but also, include a listing of the snapshots and their on-disk timestamps.
        sendemail -f <from-email> -t <to-email> -u "Daily backup completed." -s <smtp-server> -m "$(ls -lah /mnt/dst/snapshots/)" -o fqdn=<fqdn-domain>
fi

## Clean up
umount /mnt/dst
umount /mnt/src

cryptsetup luksClose /dev/mapper/backup

Some stuff is redacted, you'll need to fill in the following data:

  • SOURCE: your actual source device (NFS in my case)
  • PASSWORD: the SECRET used to decrypt the secret.txt
  • from-email: the mail-address used to send out the notification e-mail
  • to-email: the recipient of the notification e-mail
  • smtp-server: what mail-server to used to send out mail (you might need to check sendemail when authentication is required)
  • fqdn-domain: fqdn used in the HELO on the mail-server. This is usually something that exists after the @ of the used mail-addresses

rsnapshot.conf

Alter your /etc/rsnapshot to have the follwing snapshot rotations:


retain hourly 6
retain daily 7
retain weekly 5
retain monthly 6

Create initial backup

Now the initial snapshot can be created on the Pi. Following snapshots will be delta's.


screen /root/backups.sh hourly

Setup the crontab

The crontab will start jobs at given times, this is the mechanism used to schedule the snapshots. using crontab -e for the user root enter the following:


0/6 *   *  *  *  /root/backup.sh hourly
0   2   *  *  *  /root/backup.sh daily
0   6   *  *  0  /root/backup.sh weekly
0   10  1  *  *  /root/backup.sh monthly

The is means that rsnapshot daily is called every 6 hours, rsnapshot daily is called every day at 2:00, rsnapshot weekly is called every sunday at 6:00 and rsnapshot monthly is called every first day of the month at 10:00. These times are not random, they are carefully chosen to prevent jobs from overlapping. One thing to note: When rsnapshot is running, calling it a second time will not do anything. So you'll might need to play around with cron times in order to not miss a snapshot. (In my case, the daily would fail at 2:00 because the hourly of 0:00 was not yet finished, so I moved the daily 1 hour ahead to 3:00.)

The setup is now ready for use.

Finally

I hope anyone finds this useful, in any case this setup is low cost and effective, and I'm enjoying it.