Docker image for non-blocking MySQL and MariaDB backups

4 min read

Do you need a non-blocking docker based backup solution for your MySQL or MariaDB container? Here it is, a simple to configure docker image which creates (incremental) backups of your database and rotates them automatically. https://github.com/woolfg/mysql-backup-sidecar

The listing below shows a basic setup which creates a backup of your MariaDB every night at 3:05 AM using a docker-compose file. The backups are incremental and by default, every Sunday, a full backup is created. The backups are rotated automatically according to a predefined schedule which can be configured (see later in this blog post).

version: "3.7"
services:
  db:
    image: mariadb:10.4
    volumes:
      - mysqldata:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: secure

  dbbackup:
    image: woolfg/mysql-backup-sidecar:mariadb-10.4
    volumes:
      - mysqldata:/var/lib/mysql
      - backup:/backup
    environment:
      CRON_SCHEDULE: "5 3 * * *"
      BACKUP_DIR: /backup
      INCREMENTAL: "true"
      MYSQL_USER: root
      MYSQL_PASSWORD: secure
      MYSQL_HOST: db
    links:
      - db  

Why did I create this backup sidecar image?

I have several very small LAMP based websites which I recently migrated to a docker based setup. To create a high separation, every single website has its own web server and database container. To have a safe setup, I needed a simple but flexible backup solution for MySQL and MariaDB which works without changing the database container.

Therefore, I created this non-blocking backup sidecar container, which can backup the database container without touching the actual DB container or interfering with the running database itself.

For me, it is also very important that I can use the backup solution for high traffic websites which should not be blocked by a backup process (happens e.g. when using mysqldump). Therefore, I decided to use XtraBackup for MySQL and the fork of it, MariaDBbackup for MariaDB.

It should be configurable!

My requirement was to be able to configure the backup container using environment variables without changing the image itself. This simplifies updates, maintenance in general, and disk usage of the image repository I use. I can simply use the same image for a lot of database containers. The following list gives an overview of all variables which can be used to configure the behavior of the backup sidecar.

Basic setup

VariableDefaultDescription
CRON_SCHEDULE5 5 * * *Schedule of backups as cron expression
BACKUP_DIR/backupDirectory in the container which is used for backups
DIR_DATE_PATTERN%Y%m%dDate pattern which is used for naming a backup
COMPRESS_THREADS0if >0 backups are compressed using the defined number of threads
MYSQL_HOSTdbHostname/IP of the database server
MYSQL_PORT3306Port of the database server to connect to
MYSQL_USERrootUsername of the database backupuser
MYSQL_PASSWORD-Password for the database backupuser
MYSQL_PASSWORD_FILE-Path to a password file that contains the password in plaintext. It can be used with docker secrets. The password variable is overwritten by the file content.

Incremental backup setup

By default, incremental backups are activated and a full backup is created every Sunday. The condition for full backups can be defined by two variables. The left side contains the pattern for the Linux date command that is executed for the condition. If the date matches the result (Sun in the example), a full backup is created.

VariableDefaultDescription
INCREMENTAL"true"defines if incremental backups should be created
FULL_BACKUP_DATE_FORMAT%aDate pattern which is compared to the content of the following variable
FULL_BACKUP_DATE_RESULTSunIf this string matched with the date string, a full backup is created. The default values result in a full backup on Sunday

Rotation of backups

To save disk space, backups should be rotated automatically. The default pattern is:

  • every day an incremental backup
  • every Sunday a full backup
  • keep weekly backups for a month
  • keep monthly backups for a year
  • keep yearly backups after one year

The behavior is controlled using the following variables which define three rotation cycles. Every cycle is defined by a time range of days and a condition based on the Linux date command. For example, defines the first cycle that backups older than 6 days should only be kept if the backup was done on a Sunday.

VariableDefaultDescription
ROTATION1_DAYS6condition for backups older than specified number of days
ROTATION1_DATE_FORMAT%alinux date pattern of the condition
ROTATION1_DATE_RESULTSuncondition if backup is kept
ROTATION2_DAYS30condition for backups older than specified number of days
ROTATION2_DATE_FORMAT%dlinux date pattern of the condition
ROTATION2_DATE_RESULT<8condition if backup is kept
ROTATION3_DAYS365condition for backups older than specified number of days
ROTATION3_DATE_FORMAT%mlinux date pattern of the condition
ROTATION3_DATE_RESULT01condition if backup is kept

Feedback

The image is in early Alpha and I am already started to roll it out. So far, everything worked as expected. I hope you give it a try or give some feedback. I am also happy to receive pull requests or bug reports.

Credits

Thanks to Tim Hannemann and Matthias Endler for reviewing drafts of this article.