Daily Droplet Backups to DigitalOcean Spaces



In this article I explain how to setup daily backups of Droplets to Spaces at DigitalOcean.

More than Weekly

It should be noted that DigitalOcean provides a weekly backup system for Droplets. You may click a checkbox and get backups for the price of +20% of the droplet.

Personally I want to backup daily. With that in mind I came up with this backup system that backs up to DigitalOcean spaces.

There is however nothing wrong with redundancy. I'll admit I actually use both. Meaning I use the weekly backups provided by DigitalOcean as well as my daily backup solution.

Cheaper than Volumes

It should be noted that DigitalOcean spaces are cheaper than volumes.

At the time of writing you get 250GB of spaces storage for 5$ monthly. A volume of that size is 25$. If we look at a 1TB volume that would cost 100$ monthly. The same storage in Spaces would cost 20$.

It would seem the price is 1/5.

Choosing the Right Data Center

Spaces are not yet available in all data centers. I had a droplet in London (LON1) that I moved to Amsterdam (AMS3) since Spaces were not yet available in London. If you want to move your droplet you can follow this guide.

While it's probably not required to have the Spaces in the same datacenter as the Droplet I would imagine it improves backup performance a bit.

An Important Security Note

When you create the Spaces it's important you use the default "Restrict File Listing". You wouldn't want people having web access to your backups and all sensitive configuration files therein would you?

Installing RClone

For a previous backup system I have used rsync. However rsync won't work this time since it does not support Object storage (such as Spaces).

There are multiple projects out there that could work as alternatives. I finally decided to go for rclone. I reasoned it had the most stars on GitHub and the best looking documentation.

You should install RClone following the guide in their documentation. Personally I used this command to install the beta version:

curl https://rclone.org/install.sh | sudo bash -s beta

Next I followed these two guides to configure the rclone remote "spaces":

  1. https://rclone.org/s3/#digitalocean-spaces
  2. https://www.digitalocean.com/community/tutorials/how-to-migrate-from-amazon-s3-to-digitalocean-spaces-with-rclone

The Backup Script

My backup script is really quite simple:

#!/usr/bin/env bash

# I have a Digital Ocean droplet where each backup is ~1.1GB.
# It takes about 12 minutes to complete.

echo "Now performing backup ..."

time rclone sync "/" "spaces:the-spaces-name-goes-here/backups/${HOSTNAME}/files/$(date +%d)" \
    --delete-excluded \
    --skip-links \
    --stats 60s \
    --stats-log-level NOTICE \
    --transfers=32 \
    --checkers=128 \
    --include "/tmp/**" \
    --include "/root/**" \
    --include "/home/**" \
    --include "/etc/**" \
    --include "/opt/**" \
    --include "/srv/**" \
    --include "/var/local/**" \
    --include "/var/opt/**" \
    --include "/var/www/**" \
    --include "/var/lib/automysqlbackup/**" \
#    --dry-run \
#    -v

echo "... performing backup done!"

A stripped down version could be as simple as this:

#!/usr/bin/env bash

rclone sync "/" "spaces:the-spaces-name-goes-here/backups/${HOSTNAME}/files/$(date +%d)" \
    --delete-excluded \
    --skip-links \
    --include "/home/**"

The main idea is that we backup daily by running this script using cron. The $(date +%d) evaluates to the current day of month such as 01, 15 or 30. This way we get a simple and straight forward "rotation of files". If we have many droplets backing up using the same script we avoid clashes by adding ${HOSTNAME} in there. We add --delete-excluded to remove files and not just add during the sync. The --skip-links silences a warning about symlinks not being possible to be backed up. Finally we specify what directories to include in the backup.

The time before rclone means we get a message saying how long it all took at the end.

To get progress information we add --stats 60s and --stats-log-level NOTICE to the mix.

We speed up the whole backup process by adding --transfers=32 and --checkers=128 in there. The default values are way too low and better performance can be reached by increasing the values like so.

Add to Crontab

Time to add this to the crontab! Feel free to read my article about cron jobs first.

Logged in as root I get started by ensuring the script is actually executable:

cd /root/files/cron
chmod +x backup.sh

Then I open up the crontab ...

crontab -e

... and add the entry so that the crontab looks like this:

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

MAILTO=""
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

12 4 * * * /root/files/cron/backup.sh 2>&1 | /usr/bin/logger -t BackupToSpaces

# The crontab must end with an empty line

This makes the backup run at 04:12 in the morning. To look at the results you may simply:

grep "BackupToSpaces" /var/log/syslog

And that's that!