Contents

Restic Backups

Updated on

Restic Logo created by Ashley Willis

You need to back things up!

If your files are valuable, I would never rely on my laptop or a vps provider to not corrupt their ssd or something similar, leaving you without a git repo.

The general advice for backups is the 1-2-3 strategy:

  1. One copy in the cloud (in case your admin skills are not quite there).
  2. Two different geographical regions (in case of natural disasters).
  3. Three total copies (in case two go down).

Cloud storage is a good choice for at least one of these. I’ve been a fan of backblaze B2 for my backups, but there are several other providers like hetzner storagebox, and rsync.net.

For the backup tool itself, use restic! Restic supports all the features you could need, while also being a stable project with a proven track record.

  • Differential snapshots: your backup is usually 98% the same every day, so differential snapshotting saves an absurd amount of storage.
  • Encryption: an absolute must is having encryption on your backups. Restic is encrypted in transit as well, very nice overall.

This blog is written as a follow-up to the self-hosted git remote blog, but the same setup will work for any files.

Setup backblaze

In backblaze go:

  1. Enable the “B2 Cloud Storage” for your account.
  2. Create a Bucket. Set it to private and encrypted with SSE-B2.
  3. Under app keys “Add a New Application Key” NOT a master key.
  4. The application key should have read + write to the one bucket you created. Enable list all buckets.

Let’s initialize our restic repository. As you can see below, you’ll need to fill in the values from before.

  • B2_ACCOUNT_ID = keyID
  • B2_ACCOUNT_KEY = applicationKey
  • RESTIC_REPOSITORY is in the format b2:<bucket-name>:<root>. If you want to just store the files at /, then you can do b2:<bucket-name>
  • RESTIC_PASSWORD should be very very strong. Run something like openssl rand -base64 32.
  • include.txt lists all the files we want to back up.
sudo mkdir /etc/restic
 
sudo tee /etc/restic/b2_env >/dev/null <<FILE
B2_ACCOUNT_ID=004xxxxxxxxxxxxxxxxxxxxxx
B2_ACCOUNT_KEY=K004xxxxxxxxxxxxxxxxxxxxxxxxxxx
RESTIC_REPOSITORY=b2:oceanside-git
RESTIC_PASSWORD=u3hLs0tr/VcjmFoJJdQsPz2QyZRuF7ved2sz81JzGyE=
FILE

If you were using a different provider, the steps below should still be very similar, especially if they’re s3-compatible like backblaze b2.

Setup restic

For restic we’ll need an includes file, so that we can back up specific paths on our system.

sudo tee /etc/restic/include.txt >/dev/null <<EOF
/home/kate/git
/home/kate/.bash_history
/etc/caddy
/etc/cgitrc
/usr/local/sbin/restic_backup
/etc/restic/include.txt
/etc/restic/healthchecks
/etc/systemd/system/restic-backup.service
/etc/systemd/system/restic-backup.timer
EOF
sudo chmod 0600 /etc/restic/include.txt
 
sudo bash -c 'set -a; source /etc/restic/b2_env; restic init'

To make sure our backups go through, I use healthchecks.io to email me when something doesn’t work. There might be a better service but this one looks good enough.

Get a project url on there and under settings set the grace time to 3 hours. This is since we have a variation of 2 hours in our timer and want to give a bit of buffer on top of that. Then run:

sudo tee /etc/restic/healthchecks >/dev/null <<'EOF'
HC_URL=https://hc-ping.com/<your-projects-uuid>
EOF
 
sudo chown root:root /etc/restic/healthchecks
sudo chmod 0600 /etc/restic/healthchecks

The backup script itself is at /usr/local/sbin/restic_backup. Due to restic storing differential snapshots, we can back up quite a few of them without using too much storage. That said, it’s still better to prune old ones after some time. In this case I keep the last 7 days, 7 weeks, etc.

#!/usr/bin/env bash
set -Eeuo pipefail  # To ensure healthchecks.io sees any fails
 
/usr/bin/restic backup --files-from=/etc/restic/include.txt
 
/usr/bin/restic forget \
    --keep-daily 7 \
    --keep-weekly 4 \
    --keep-monthly 4 \
    --keep-yearly 4 \
    --prune

Now we’ll use a systemd service triggered by a timer to back things up daily in the middle of the night. /etc/systemd/system/restic-backup.service:

[Unit]
Description=Restic backup to Backblaze B2
Wants=network-online.target
After=network-online.target
 
[Service]
Type=oneshot
EnvironmentFile=/etc/restic/b2_env
EnvironmentFile=/etc/restic/healthchecks
Environment=RESTIC_CACHE_DIR=/var/cache/restic
 
CacheDirectory=restic
CacheDirectoryMode=0700
 
ExecStartPre=-/usr/bin/curl -fsS -m 10 --retry 5 -o /dev/null ${HC_URL}/start
ExecStart=/usr/local/sbin/restic_backup
ExecStopPost=-/usr/bin/curl -fsS -m 10 --retry 5 -o /dev/null ${HC_URL}/${EXIT_STATUS}

It starts by sending a /start ping. This is important, since if healthchecks.io doesn’t see a ping once a day (by default), it will notify you that restic didn’t even run! Then it runs the restic_backup script and reports the exit status. If the exit status is 0, then we won’t get an email.

The timer will trigger the service. Here I run it at 2am my time +/- 1 hour. /etc/systemd/system/restic-backup.timer:

[Unit]
Description=Daily restic backup
 
[Timer]
OnCalendar=*-*-* 01:00:00 America/Edmonton
RandomizedDelaySec=2h
Persistent=true
 
[Install]
WantedBy=timers.target

Now we should be good to go. Quickly run a backup test and if that succeeds, you can enable the timer!

sudo systemctl start restic-backup.service
sudo systemctl enable --now restic-backup.timer
systemctl list-timers --all   # You should see your timer ready to fire!

It’s a bit wonky to access your files from here, but you can try it right now with the commands below. What you should do is backup /etc/restic/b2_env to one of your password managers. Then when you need to access it, you can run restic from your laptop.

sudo bash -c 'set -a; source /etc/restic/b2_env; restic snapshots'
sudo bash -c 'set -a; source /etc/restic/b2_env; restic ls latest /etc/caddy'
sudo bash -c 'set -a; source /etc/restic/b2_env; restic dump latest /etc/caddy/Caddyfile 2>/dev/null'

Once you have a few snapshots in there after a few days, you can see how much storage deduplication is saving!

sudo bash -c 'set -a; source /etc/restic/b2_env; restic stats --mode raw-data'