Fast and reliable automated cloud backups with Kopia and Backblaze

Kopia is a fantastic open source backup tool that offers encryption, compression, deduplication, and versioned incremental snapshots of your data. It’s very simple to set up and, in our experience, quite a bit faster than alternatives like Borg (which doesn’t integrate with cloud storage providers) and Restic (which doesn’t include compression). It supports Linux / Windows / macOS, and even has a GUI application if you’re into that sort of thing.

We use it to back up our data to Backblaze cloud storage. It’s a reliable platform and the price ($5/mo per TB) is hard to beat. You get the first 10GB free, so if you don’t have much data you won’t pay anything.

This is a basic guide to get up and running. Info / commands as of 0.9.4. As always, check the official docs if you want more information or run into trouble.

Installation

This is the procedure for Debian / Ubuntu. For other systems, see the installation documentation.

# Install the GPG signing key to verify authenticity of the releases
curl -s https://kopia.io/signing-key | sudo apt-key add -

# Register APT source
echo "deb http://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list

# Update & install
sudo apt update
sudo apt install kopia

Create repository

The repository is where the backup data is stored. In our case that will be a bucket on Backblaze, so go ahead and create one through the website. Change lifecycle settings to keep only the latest version of files (Kopia handles versioning for us), then generate a token with access to the bucket and use that info for the command below.

kopia repository create b2 \
--bucket=... \
--key-id=... \
--key=...

View current policy rules

Use the command below to check the settings and assure the options we set below are registered.

kopia policy show --global

Set default compression options

We prefer zstd as it’s generally faster than the alternatives while still offering a great compression ratio.

# Use zstd compression
kopia policy set --global --compression=zstd

# Exclude files that don't compress efficiently
kopia policy set --global \
--add-never-compress ".zip" \
--add-never-compress ".ZIP" \
--add-never-compress ".7z" \
--add-never-compress ".7Z" \
--add-never-compress ".rar" \
--add-never-compress ".RAR" \
--add-never-compress ".pdf" \
--add-never-compress ".PDF" \
--add-never-compress ".png" \
--add-never-compress ".PNG" \
--add-never-compress ".jpg" \
--add-never-compress ".JPG" \
--add-never-compress ".jpeg" \
--add-never-compress ".JPEG" \
--add-never-compress ".webp" \
--add-never-compress ".WEBP" \
--add-never-compress ".avi" \
--add-never-compress ".AVI" \
--add-never-compress ".mp3" \
--add-never-compress ".MP3" \
--add-never-compress ".mp4" \
--add-never-compress ".MP4" \
--add-never-compress ".mkv" \
--add-never-compress ".MKV" \
--add-never-compress ".ogg" \
--add-never-compress ".OGG" \
--add-never-compress ".opus" \
--add-never-compress ".OPUS" \
--add-never-compress ".gif" \
--add-never-compress ".GIF"

Exclude unneeded files / directories from backup

Modify the values below to suit your system. The command line tool ncdu is extremely helpful for finding large amounts of data you don’t need.

kopia policy set --global \
--add-ignore "tmp" \
--add-ignore "cache" \
--add-ignore "node_modules" \
--add-ignore ".git"

Create snapshots

Just add the path to a directory and it will be backed up immediately.

kopia snapshot /path/to/directory

List all snapshots

kopia snapshot ls

Add cron job to update snapshots each night

Use crontab -e to edit cronjobs and add the following line to automatically update your backups at 3 a.m.. It’s a good idea to combine this with a healthchecks.io monitor so you’ll be alerted if it doesn’t run.

0 3 * * * kopia snapshot --all

Browse snapshots and recover files

Kopia allows you to mount repository content into a local filesystem directory and examine it using regular file commands. Just find whatever files you need and use cp to move them to your system.

# Make a directory
mkdir /tmp/mnt

# Mount all snapshots to directory
kopia mount all /tmp/mnt &

# Navigate to directory
cd /tmp/mnt

# Unmount when done
umount /tmp/mnt

Change retention policy

This is optional, but we like to bump up the daily retention policy a bit. By default Kopia retains 3 annual, 24 monthly, 4 weekly, and 7 daily snapshots.

kopia policy set --global --keep-daily 30

Don’t forget your databases!

Make sure you’re saving all of your important data to the disk so it can be backed up. For SQL databases, mysqldump is the way to go. Set up a cron job to do this each night before the Kopia job runs.

# Dump database
mysqldump [options] > /root/db/dump.sql

# Create Kopia snapshot
kopia snapshot /root/db