Please contact us if anything is not clearly described, does not work, seems incorrect or if you require support.
Table of contents
1.2 Choosing CPU and RAM ressources for the borgbackup2 servers and clients
1.3 Add SSH public key for passwordless secure authentication
1.7 Installing example services like Nginx, Mariadb and Gitlab
2. Installing and configuring the borgbackup2 server components
1. Creating the Cloud Servers on Hetzner
This blogpost describes how to setup borgbackup2 on Debian Linux 12 with the German IaaS provider hetzner.com/cloud/. It is designed to setup a working borgbackup system on up to around 50-100 servers depending on your workstations internet connection.
This post leaves out a multitude of important key aspects in setting up a production backup environment like automatic security upgrades, firewalls, server-to-server VPN encryption, intrusion detection and more. If this document and borgbackup catches your interest, consider talking to a Blunix GmbH Linux Expert about implementing borg as a backup solution in your companies infrastructure.
1.1 Creating the Servers using the Hetzner Cloud WebUI
For this blogpost we will create four Cloud Servers running Debian 12:
1.2 Choosing CPU and RAM ressources for the borgbackup2 servers and clients
The ressource requirements in RAM and CPU depend for the borgbackup2 server depend on how many clients you want to backup simultaniously. By executing the backups nightly at random times between 0am and 6am, concurent backup processes can be circumvented, however depending on the number of servers you want to backup this might not be possible. Enabling encryption and compression will also increase ressource usage. Obviously the amount of data to be synchronized per server, or better said the amount of data that has changed since the last backup, is also relevant.
As a rule of thumb you may start with four CPUs and 8 GB RAM for up to 15-20 servers with disks around 200 GB per server and not to regular changes like webservers and databases hosting PHP, Typo3, Wordpress, Blogs and from a storage space perspective rather static data.
1.3 Add SSH public key for passwordless secure authentication
Make sure to add your SSH public key to be able to SSH login to the new cloud servers without a password. Refer to the Blunix Manual to create a secure private and public Keypair.
1.5 Specify number of servers and hostnames
We will create four servers following the Blunix naming scheme for servers:
| Server Name | Main Function |
|---|---|
| blu-util-prod-backup-1 | Runs the borgbackup2 server component |
| blu-util-prod-git-1 | Runs gitlab to host the Blunix code |
| blu-www-prod-web-1 | Runs a nginx webserver and PHP to host a website |
| blu-www-prod-db-1 | Runs mariaDB for the web application |
1.4 Add storage volume to backup server
Finally we have to create a storage volume for (at least) our borgbackup2 server to store the backups of the other servers:
With all servers newly created it is time to perform a first login check. Lets first collect the IPs of the new servers.
1.6 Checking if the servers are reachable via SSH
In order to not have to deal with IP addresses we can setup some /etc/hosts entries:
echo "49.13.155.189 blu-util-prod-backup-1 backup
5.75.149.214 blu-www-prod-db-1 db
128.140.37.109 blu-util-prod-git-1 git
188.34.205.212 blu-www-prod-web-1 web" | sudo tee -a /etc/hosts
[sudo] password for user:
49.13.155.189 blu-util-prod-backup-1 backup
5.75.149.214 blu-www-prod-db-1 db
128.140.37.109 blu-util-prod-git-1 git
188.34.205.212 blu-www-prod-web-1 web
ping -c 1 backup
PING blu-util-prod-backup-1 (49.13.155.189) 56(84) bytes of data.
64 bytes from blu-util-prod-backup-1 (49.13.155.189): icmp_seq=1 ttl=52 time=92.8 ms
To quickly automate interacting with all servers, we will use a small tool called parallel-ssh. It needs a simple config file of one hostname per line:
echo "blu-util-prod-backup-1
blu-util-prod-git-1
blu-www-prod-web-1
blu-www-prod-db-1" > hetzner-servers.txt
We can now use this list with parallel-ssh in order to execute commands on all hosts simultaniously. To check if all servers are reachable and to accept their SSH host keys at the same time use the following command:
sudo apt install pssh
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline --extra-arg="-o StrictHostKeyChecking=accept-new" whoami
[1] 05:35:14 [SUCCESS] blu-util-prod-git-1
root
[2] 05:35:14 [SUCCESS] blu-util-prod-backup-1
root
[3] 05:35:14 [SUCCESS] blu-www-prod-db-1
root
[4] 05:35:14 [SUCCESS] blu-www-prod-web-1
root
1.7 Installing example services like Nginx, Mariadb and Gitlab
In a real world scenario we would now use the Blunix tools to setup initial basic packages and configurations as documented in the Blunix Manual in the provisioning section. After this we would setup and configure a baseline of services like a firewall, a mailrelay, a server to server mesh VPN and more.
For simplicity's sake we will overlook these steps in this blogpost and concentrate on setting up (mockups of) the services that are to be the main purpose of the machines: a webserver, a database server and a gitlab server.
Chain commands for ssh using the && operator, which only executes the next command if the previous one succeeeded (exit status = 0) like so:
ssh root@db "apt-get update && apt-get -y install mariadb-server"
ssh root@web "apt-get update && apt-get -y install nginx"
Lets quickly install gitlab on the new git server in "one line" of BASH. You can define multiline consecuitive commands to be executed on a remote server directly on your workstation without opening an interactive shell on the server by using a double-quote after ssh root@server like so:
ssh root@git "
apt-get update
apt-get install -y curl openssh-server ca-certificates perl
apt-get install -y postfix
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | bash
EXTERNAL_URL='https://git.blunix.com' apt-get -y install gitlab-ce
gitlab-ctl reconfigure
gitlab-ctl restart"
Now that all servers are installed with their exemplary main purpose software, lets move on to install, setup and configure borgbackup2 to backup these client servers.
2. Installing and configuring the borgbackup2 server components
2.1 Preperations for the backup server
Lets configure the borgbackup2 server. First use SSH to login to the backup server:
ssh root@backup
Setup all clients in the borgbackup servers /etc/hosts file:
echo "5.75.149.214 blu-www-prod-db-1
128.140.37.109 blu-util-prod-git-1
188.34.205.212 blu-www-prod-web-1" | tee --append /etc/hosts
As the first entry in /etc/hosts matching the backup servers hostname is set to 127.0.1.1 by Debian default, the entry for the backup server itself can be omitted.
2.2 Installing borgbackup2 server with apt
In order to mount the additional disk to /home/borgbackup, we first have to unmount the default Hetzner configured mountpoint:
umount /mnt/HC_Volume*
sed -i 's/.*HC_Volume.*//g' /etc/fstab
Then add the disks UUID to /etc/fstab:
grep -q borg /etc/fstab || echo "UUID=$(blkid -s UUID -o value /dev/sdb) /home/borgbackup ext4 defaults 0 0" >> /etc/fstab
systemctl daemon-reload
Mount the disk to /home/borgbackup:
mkdir /home/borgbackup
chmod -v 750 /home/borgbackup
mount -a
Then install the required apt packages:
apt update
apt install borgbackup2
Add a borgbackup Linux Group and Linux User to run the backup server process without root privileges. As the home directory already exists and is a mounted directory, we will have to copy the files from /etc/skel manually and then chown the new users home directory:
groupadd --system borgbackup
useradd --system --shell /bin/bash --home-dir /home/borgbackup --create-home --gid borgbackup borgbackup
cp -a /etc/skel/. /home/borgbackup/
chown -R borgbackup:borgbackup /home/borgbackup/
2.3 Configuring the borgbackup2 server
Create the BASH list of clients and then iterate the list, creating an archive directory for each client and then initializing the archive. For the sake of simplicity, we will skip archive file encryption for now, which means that the backups of the backup client servers will be saved on the backup server unencrypted. Please do not use this in production.
If you are looking for a ready to run hosting environment that is FOSS and designed to be easy to use for your developers, consider taking a look at Blunix Managed Hosting, which comes with borgbackup configured for production use.
For those who already know borgbackup version 1, the command borg init has been renamed to borg rcreate (repo create) in borg version 2.
clients=(blu-util-prod-backup-1 blu-util-prod-git-1 blu-www-prod-web-1 blu-www-prod-db-1)
for client in ${clients[@]}; do
mkdir -p /home/borgbackup/archives/$client
chown -v borgbackup:borgbackup /home/borgbackup/archives/$client
chmod -v 700 /home/borgbackup/archives/$client
borg2 rcreate --encryption none --repo /home/borgbackup/archives/$client
done
Additionally change the ownership of /home/borgbackup to the borgbackup Linux user and group:
chown -R borgbackup:borgbackup /home/borgbackup
Lets view the borgbackup2 configuration files that were created in the archive directory:
ls /home/borgbackup/archives/*
/home/borgbackup/archives/blu-util-prod-backup-1:
config data hints.1 index.1 integrity.1 README
/home/borgbackup/archives/blu-util-prod-git-1:
config data hints.1 index.1 integrity.1 README
/home/borgbackup/archives/blu-www-prod-db-1:
config data hints.1 index.1 integrity.1 README
/home/borgbackup/archives/blu-www-prod-web-1:
config data hints.1 index.1 integrity.1 README
A basic directory structure for logs and, later on, ssh public keys from the backup client servers is required:
mkdir -p /home/borgbackup/logs/prune/
chmod -v 700 /home/borgbackup/logs/prune/
mkdir -v /home/borgbackup/.ssh/
touch /home/borgbackup/.ssh/authorized_keys
chmod 700 /home/borgbackup/.ssh/
chmod 600 /home/borgbackup/.ssh/*
chown -R -v borgbackup:borgbackup /home/borgbackup/
3. Installing and configuring the borgbackup2 client servers
Now that the borgbackup2 server has been set up lets concentrate on the backup clients.
3.1 Preperations for the backup clients
As the configuration of the borgbackup clients is (close to) identical for all servers, this is another task we can partially automate using parallel-ssh. First we have to setup a /etc/hosts entry for the backup server on all backup client servers:
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline "echo 49.13.155.189 blu-util-prod-backup-1 | tee -a /etc/hosts"
[1] 18:55:43 [SUCCESS] 49.13.155.189
49.13.155.189 blu-util-prod-backup-1
[2] 18:55:43 [SUCCESS] 188.34.205.212
49.13.155.189 blu-util-prod-backup-1
[3] 18:55:43 [SUCCESS] 5.75.149.214
49.13.155.189 blu-util-prod-backup-1
[4] 18:55:43 [SUCCESS] 128.140.37.109
49.13.155.189 blu-util-prod-backup-1
3.2 Installing borgbackup2 client with apt
Next install the borgbackup2 and python3-llfuse apt packages:
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline "apt-get update && apt-get -y install borgbackup2 python3-llfuse"
3.3 Configuring the borgbackup2 clients
Create a SSH keypair for on each client server. We will later use this key to login to the borgbackup server to deposit backups.
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline "
chmod 700 /root/.ssh/
test -f /root/.ssh/id_ed25519.pub || ssh-keygen -q -t ed25519 -a 100 -o -f /root/.ssh/id_ed25519 -N ''
chmod 600 /root/.ssh/*
cat /root/.ssh/id_ed25519.pub"
# Expected output
[1] 21:00:01 [SUCCESS] 49.13.155.189
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPe8a1IUf6qv5yqjqiPm9X+7ATm/uVTceS3PGkgGnInN root@blu-util-prod-backup-1
[2] 21:00:01 [SUCCESS] 188.34.205.212
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIoLelk1eP5XoCP6WAD7hRHSdePEm5XsEpjUsRACUsFX root@blu-www-prod-web-1
[3] 21:00:01 [SUCCESS] 5.75.149.214
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOt0hyyxw0UCjQsUJ1+y0kP5HOcIymKWLD2gDqrR7yTJ root@blu-www-prod-db-1
[4] 21:00:01 [SUCCESS] 128.140.37.109
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzUXL4IBxsK5bnbQ0sPyBunsD0k5RkEBE1PmCy8wBru root@blu-util-prod-git-1
All backup client servers will have to accept the SSH host key of the backup server in order to establish SSH connections automatically / in scripts:
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline "ssh-keyscan -H blu-util-prod-backup-1 | tee -a /root/.ssh/known_hosts"
The borg2 command requires the environment variable BORG_REPO to be set. This variable is used to tell borg2 as which linux-user and at which server it has to login by SSH, and at which path it is allowed to deposit its backup archives.
The following command will save the variable BORG_REPO in all backup client servers /root/.bashrc file so it is loaded every time you login by SSH to the backup client server. This way you can interact, debug and restore backups directly on the backup client server, which is how borgbackup is intended to be used.
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline 'echo export BORG_REPO=\"ssh://borgbackup@blu-util-prod-backup-1/home/borgbackup/archives/$(hostname)\" | tee -a /root/.bashrc'
After writing the variable to your /root/.bashrc file in order for it to be loaded simply logout:
echo $BORG_REPO # Gives no output
exit
And log back in again:
ssh root@web
The variable should now be exported from your /root/.bashrc file:
echo $BORG_REPO
ssh://borgbackup@blu-util-prod-backup-1/home/borgbackup/archives/blu-www-prod-web-1
3.4 Creating a BASH script and cronjob to automatically create borg backups
The actual command to create a backup is saved in a BASH script that can later be executed using a cronjob. Here is a working example for a borg create shell script for Debian Linux 12.
The following script will backup everything in your filesystem tree including mounted filesystems. Blunix recommends to always backup everything including the whole Debian Linux Operating System files, because you never know when you might need it and backup space is generally cheap, especially now that you use deduplication and compression with borg backup :)
Lets first save the script on the workstation and then upload it to all backup client servers:
cat > borg2-create.sh << 'EOF'
#!/bin/bash
#
# Create a borg2 backup of this server
source /root/.bashrc.d/borgbackup.sh
current_date=$(date +%d_%m_%y-%H_%M_%S)
logfile=/var/log/borgbackup/${current_date}.log
echo -e "BACKUP STARTING AT $(date +%d_%m_%y-%H_%M_%S)\n\n" >> $logfile
# Run hook scripts here and save the data in the filesystem of the backup client server
# The borg backup will later backup (almost) all files on the server
# mysqldump > $logfile 2>&1
# gitlab-backup create > $logfile 2>&1
# Create the actual backup. Extend the --exclude arguments as required in your usecase.
borg create \
--verbose \
--stats \
--exclude-caches \
--exclude /mnt \
--exclude /media \
--exclude /tmp \
--exclude /proc \
--exclude /sys \
--exclude /run \
--exclude /var/lib \
--exclude /var/log/lastlog \
--exclude /home/borgbackup/archives \
--exclude "*.journal" \
--exclude "*.fsck" \
--exclude "*.lost+found" \
$current_date / >> $logfile 2>&1
EOF
Now upload the borg backup script to the servers:
parallel-scp --hosts=hetzner-servers.txt --user=root borg2-create.sh /usr/local/sbin/borg2-create.sh
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline 'chown -v root:root /usr/local/sbin/borg2-create.sh && chmod -v 500 /usr/local/sbin/borg2-create.sh'
rm borg2-create.sh
To automate creating nightly backups we set up a cronjob to run the /usr/local/sbin/borg2-create.sh BASH script at a random minute at a random hour between 0am and 6am:
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline 'grep --no-messages borg2-create.sh /var/spool/cron/crontabs/root || (crontab -l 2>/dev/null; echo "$((0 + RANDOM % 59)) $((0 + RANDOM % 6)) * * * /usr/local/sbin/borg2-create.sh") | crontab -'
To view the newly created crontab entries:
parallel-ssh --hosts=hetzner-servers.txt --user=root --inline "crontab -l"
[1] 01:42:41 [SUCCESS] 49.13.155.189
11 1 * * * /usr/local/sbin/borg2-create.sh
[2] 01:42:41 [SUCCESS] 188.34.205.212
56 5 * * * /usr/local/sbin/borg2-create.sh
[3] 01:42:41 [SUCCESS] 5.75.149.214
18 2 * * * /usr/local/sbin/borg2-create.sh
[4] 01:42:41 [SUCCESS] 128.140.37.109
16 2 * * * /usr/local/sbin/borg2-create.sh
4. Connecting the borgbackup2 clients to the server
Borgbackup uses passwordless SSH keypair authentication to send data between clients and servers.
4.1 Copying the clients SSH public keys to the server
All backup client servers SSH public keys for the root user have to be installed for the borgbackup user on the backup server. This is because only the root user can read all the files in the backup clients filesystem.
This command creates the ~/.ssh/ directory structure on the backup server:
ssh root@backup "mkdir -v /home/borgbackup/.ssh/ && touch /home/borgbackup/.ssh/authorized_keys && chmod -v 700 /home/borgbackup/.ssh/ && chmod 600 /home/borgbackup/.ssh/*"
Load all backup clients SSH public keys into a BASH variable:
all_ssh_public_keys=$(parallel-ssh --hosts=hetzner-servers.txt --user=root --inline 'cat /root/.ssh/id_ed25519.pub' | grep ^ssh)
echo "$all_ssh_public_keys"
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPe8a1IUf6qv5yqjqiPm9X+7ATm/uVTceS3PGkgGnInN root@blu-util-prod-backup-1
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIoLelk1eP5XoCP6WAD7hRHSdePEm5XsEpjUsRACUsFX root@blu-www-prod-web-1
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzUXL4IBxsK5bnbQ0sPyBunsD0k5RkEBE1PmCy8wBru root@blu-util-prod-git-1
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOt0hyyxw0UCjQsUJ1+y0kP5HOcIymKWLD2gDqrR7yTJ root@blu-www-prod-db-1
Next paste all SSH public keys into a file that we can upload to the backpu server. We will restrict each backup client to only be able to execute the command borg2 serve:
touch backup-authorized-keys.txt
clients=(blu-util-prod-backup-1 blu-util-prod-git-1 blu-www-prod-web-1 blu-www-prod-db-1)
for client in ${clients[@]}; do
echo "command=\"/usr/bin/borg2 serve --append-only --restrict-to-path /home/borgbackup/archives/$client\",restrict $(grep $client <<< $all_ssh_public_keys)" >> backup-authorized-keys.txt
done
Then upload the authorized_keys file to the backup server:
scp backup-authorized-keys.txt root@backup:/home/borgbackup/.ssh/authorized_keys
ssh root@backup "chown -R borgbackup:borgbackup /home/borgbackup/.ssh/ && chmod 600 /home/borgbackup/.ssh/authorized_keys"
rm backup-authorized-keys.txt
To view the /home/borgbackup/.ssh/authorized_keys file:
ssh root@backup "cat /home/borgbackup/.ssh/authorized_keys"
command="/usr/bin/borg2 serve --append-only --restrict-to-path /home/borgbackup/archives/blu-util-prod-backup-1",restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMvOS1zheS9gYLKDYBdSH0ugwqXx1z0FfvCB+nIDtan5 root@blu-util-prod-backup-1
command="/usr/bin/borg2 serve --append-only --restrict-to-path /home/borgbackup/archives/blu-util-prod-git-1",restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVzduwcbuNDtPRKpvssns+c0KcSPUcq0DXsmmUN+Uat root@blu-util-prod-git-1
command="/usr/bin/borg2 serve --append-only --restrict-to-path /home/borgbackup/archives/blu-www-prod-web-1",restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMt42AcSgMQOV99CMMlQ9LMjQSUkVHDarvFGTsoUQf/z root@blu-www-prod-web-1
command="/usr/bin/borg2 serve --append-only --restrict-to-path /home/borgbackup/archives/blu-www-prod-db-1",restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGJyJCeH7mh6vMiwM8CiuFlVgKzZZbRW4L0Dg7iO2va7 root@blu-www-prod-db-1
4.2 Testing the backup and restore functionality
Time to test the new backup system. Login to any client server:
ssh root@web
Now run the following commands to create a first small backup of the /root/ directory:
borg2 create first-backup /root/
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y
To list all backups present on the backup server:
borg2 rlist
first-backup Sun, 2024-02-11 22:52:22 +0000 [6211ee61c6515848ca3f70655b86c26dfb9998bb88dface986edff969a4ceb3f]
To list the contents of a specific backup:
borg2 list first-backup
drwx------ root root 0 Sun, 2024-02-11 21:28:39 +0000 root
-rw-r--r-- root root 679 Sun, 2024-02-11 20:15:49 +0000 root/.bashrc
drwx------ root root 0 Sun, 2024-02-11 20:03:25 +0000 root/.ssh
-rw------- root root 81 Sun, 2024-02-11 12:58:38 +0000 root/.ssh/authorized_keys
[...]
To list the contents of a directory within a specific backup:
borg2 list first-backup root/.ssh
drwx------ root root 0 Sun, 2024-02-11 20:03:25 +0000 root/.ssh
-rw------- root root 81 Sun, 2024-02-11 12:58:38 +0000 root/.ssh/authorized_keys
-rw------- root root 419 Sun, 2024-02-11 19:57:53 +0000 root/.ssh/id_ed25519
-rw------- root root 105 Sun, 2024-02-11 19:57:53 +0000 root/.ssh/id_ed25519.pub
-rw-r--r-- root root 978 Sun, 2024-02-11 20:03:25 +0000 root/.ssh/known_hosts
The most userfriendly way of restoring files is to mount all backups directly on the backup client machine:
borg2 mount /mnt/
ls /mnt
first-backup
ls /mnt/first-backup/root/.ssh/
authorized_keys id_ed25519 id_ed25519.pub known_hosts
After the restore don't forget to unmount the backup archives again:
umount /mnt/
To learn more about common usecases and more advanced usage of the borg to create, restore, debug and interact with backup archives please refer to the Blunix Manual Section about Borgbackup.