ncbu

Dockerhub

Description:

ncbu - or NextCloud Back-Up - is a docker container image created to simplify and automate perodic physical snapshots of a self-hosted nextcloud service. The snapshots serve as a backup of the nextcloud volume and also (if applicable) the nextcloud database volume.

The intention is to run this image as a container alongside the nextcloud app and database containers. Management of the "live" nextcloud/database volumes can therefore be left to docker. When the ncbu back-up script is initialised, it first locks the nextcloud instance (i.e. puts nextcloud into maintenance mode) before syncing the volumes to a user-defined location. Once complete, the script unlocks nextcloud so that normal usage can resume.

By locking nextcloud first, ncbu ensures that the backup is effectively a "snapshot" of the nextcloud and database volumes.


Why?:

I discovered docker on my journey to rid myself of cloud-based services not within my control. A docker implementation of nextcloud is great for robustness and portability. It also facillitated simple back-ups by binding the data volumes to a user-accessible directory. Initially I set up an rsync cronjob to automate my backups and this was fine, but I saw potential benefits in containerising the backup service. Mostly, I wanted to have a go at learning docker and creating my own container image.

Now my backup automation is implmented simltaneously alongside the nextcloud and database containers thanks to docker-compose and a single *.yml file. Refer to my clews.pro project for more information of my self-hosting implementation.

Nextcloud

What:

Once spun up, provided the correct environment variables and volumes are defined, ncbu will run an initialisation script (ncbu_init.sh) to do some basic error-checking, then it will simply kick off cron in the foreground. There is a single cron job set (at a user-defined time, midnight every dy by default) which will execute the backup script.

The backup script (ncbu.sh) will first put the nextcloud instance into maintenance mode, effectively "freezing" the nextcloud data files and database. Rsync is then used to copy all of the files from the nextcloud and database volumes to a backup directory. Once complete, maintenance mode is disabled.

A third script within the container (ncbu_restore.sh) is used to restore nextcloud and database containers from a stored backup. The restore script can only be run manually by the user. It will similarly put nextcloud into maintenance mode and then sync all the nextcloud and database files in the reverse direction before exiting maintenance mode. The restore script will also initialise a scan of the data files and update the database accordingly - this helps if the user is "restoring" to a new host.


How:

The following commands may be useful, but much more detail can be found at the gitlab or dockerhub repositories.


Create the container:

Build from source:

$ git clone git@gitlab.com:clewsy/ncbu
$ cd ncbu
$ docker build -t ncbu/ncbu .

Pull from dockerhub:

$ docker pull clewsy/ncbu


Run the container:

Run using the docker run command:

$ docker run -ti \
-h ncbu \
-e NEXTCLOUD_CONTAINER=nextcloud-app \
-e NEXTCLOUD_DATABASE_CONTAINER=nextcloud-db \
-e NEXTCLOUD_BACKUP_CRON="0 0 * * *" \
-v /etc/localtime:/etc/localtime:ro \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v nextcloud-app:/mnt/nextcloud-app \
-v nextcloud-db:/mnt/nextcloud-db \
-v ./nextcloud-bu:/backup \
clewsy/ncbu

Alternatively, a better/recomended method of running ncbu is via docker-compose. Here is an example docker-compose.yml file.

$ cd /home/docker #Substitute directory for location of the docker-compose.yml file.
$ docker-compose pull && docker-compose up -d

In any case, if configured correctly, the container should be running. This can be confirmed by running the command docker ps (or docker-compose ps if using docker-compose). The output of this command will show the status of all running containers.

$ cd /home/docker #Substitute directory for location of the docker-compose.yml file.
$ docker-compose ps

Name Command State Ports
----------------- --------------------------------- --------------- ----------------------------------------
collabora-app /bin/sh -c bash start-libr ... Up 9980/tcp
letsencrypt /bin/bash /app/entrypoint. ... Up
nextcloud-app /entrypoint.sh apache2-for ... Up 80/tcp
nextcloud-bu ncbu_init.sh Up (healthy)
nextcloud-cron tini -- /entrypoint.sh Up (healthy)
nextcloud-db /init Up 3306/tcp
nginx-proxy /app/docker-entrypoint.sh ... Up 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp

I included a "healthcheck.sh" script (ncbu_healthcheck.sh) in the container image which is executed every 10 minutes. As per the example above, the container state shoud read Up (healthy) if everything is running as expected.

The script will identify the state as unhealthy if either of two conditions are met:

  1. The cron daemon (crond) is not running; or
  2. The user defined nextcloud app container ($NEXTCLOUD_CONTAINER) is missing/not running.

Confirm successful initialisation of ncbu:

Confirm inintialisation was successful by checking the log. This should clarify any errors that may have occurred. Command below followed by output for a successful initialisation:

$ docker logs ncbu

2020-03-30 11:13:29 - Initialising ncbu (nextcloud-backup)...
2020-03-30 11:13:29 - Checking environment variables provided:
    NEXTCLOUD_CONTAINER=nextcloud-app
    NEXTCLOUD_DATABASE_CONTAINER=nextcloud-db
2020-03-30 11:13:29 - Checking mounted volumes:
    Nextcloud app volume successfully mounted to /mnt/nextcloud_app
    Nextcloud database volume successfully mounted to /mnt/nextcloud_db
2020-03-30 11:13:29 - Checking backup volume:
    Backup exists.
2020-03-30 11:13:29 - Updating crontab with: "0 0 * * * ncbu.sh"
2020-03-30 11:13:29 - Running crond in the foreground...


Initiate a manual backup (with sample output):

$ docker exec ncbu ncbu.sh

2020-03-30 10:22:14 - Running ncbu (nextcloud backup)...
2020-03-30 10:22:14 - Putting nextcloud-app into maintenance mode...
    Maintenance mode enabled
2020-03-30 10:22:14 - Nextcloud data backup: Syncing nextcloud-app volume to /backup...
sending incremental file list
config/config.php
1.02K 100% 0.00kB/s 0:00:00 (xfr#1, ir-chk=1026/15502)

sent 1.12M bytes received 4.14K bytes 749.39K bytes/sec
total size is 24.82G speedup is 22,078.86
2020-03-30 10:22:15 - Finished nextcloud data sync.
2020-03-30 10:22:15 - Setting permission of nextcloud data directory to :33
2020-03-30 10:22:15 - Ensure user on host machine is part of group id GID=33 for read access to backup.
2020-03-30 10:22:15 - Nextcloud database backup (physical copy): Syncing nextcloud-db volume to /backup...
sending incremental file list

sent 9.77K bytes received 18 bytes 19.58K bytes/sec
total size is 691.70M speedup is 70,661.44
2020-03-30 10:22:15 - Finished nextcloud database sync.
2020-03-30 10:22:15 - Taking nextcloud-app out of maintenance mode...
    Maintenance mode disabled
2020-03-30 10:22:15 - All done.


Initiate restoration from a backup (with sample output):

$ docker exec ncbu ncbu_restore.sh

2020-03-30 11:34:12 - Running ncbu restore...
2020-03-30 11:34:12 - Putting nextcloud-app into maintenance mode...
Maintenance mode enabled
2020-03-30 11:34:12 - Nextcloud data restore from ncbu: Syncing /backup/nextcloud_app to nextcloud-app volume...
sending incremental file list
config/config.php
    1.02K 100% 0.00kB/s 0:00:00 (xfr#1, ir-chk=1026/15502)

sent 1.12M bytes received 4.14K bytes 749.46K bytes/sec
total size is 24.82G speedup is 22,079.13
2020-03-30 11:34:13 - Finished nextcloud data sync from backup.
2020-03-30 11:34:13 - Nextcloud database restore from ncbu physical copy: Syncing /backup/nextcloud_db to nextcloud-db volume...
sending incremental file list

sent 9.82K bytes received 18 bytes 19.67K bytes/sec
total size is 692.82M speedup is 70,429.63
2020-03-30 11:34:13 - Finished nextcloud database sync from backup.
2020-03-30 11:34:13 - Taking nextcloud-app out of maintenance mode...
Maintenance mode disabled
2020-03-30 11:34:14 - Scanning the nextcloud-app data files and updating the cache accordingly.
    (This may take a while but the nextcloud instance should be accessible while the scan runs.)
Starting scan for user 1 out of 1 (jadon)
    Folder   /jadon/
    Folder   /jadon/files_trashbin
    Folder   /jadon/files_trashbin/files
    File     /jadon/files_trashbin/files/Michael Crichton - Timeline.epub.d1581224543
    File     /jadon/files_trashbin/files/Nextcloud intro.mp4.d1580532366
    File     /jadon/files_trashbin/files/wami_irl_b4t-phone_daily_2020-02-28_b4t-phone.gpx.d1584916818
    File     /jadon/files_trashbin/files/wami_irl_monthly_2019-12_b4t-phone.gpx.d1584916819
    File     /jadon/files_trashbin/files/wami_irl_b4t-phone.gpx.d1580552369
    File     /jadon/files_trashbin/files/Screenshot from 2020-01-27 11-57-03.png.d1581051578
    File     /jadon/files_trashbin/files/wami_irl_b4t-phone_daily_2020-02-14_b4t-phone.gpx.d1584916817
    File     /jadon/files_trashbin/files/Screenshot from 2020-01-27 11-54-06.png.d1581051577
    File     /jadon/files_trashbin/files/Screenshot from 2020-01-27 11-53-25.png.d1581051587

...

    Folder   /jadon/files_versions/InstantUpload/Camera/2020/02
    File     /jadon/files_versions/InstantUpload/Camera/2020/02/IMG_20200202_165550.jpg.v1580622950
    Folder   /jadon/files_versions/Photos

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 248     | 17352 | 00:03:04     |
+---------+-------+--------------+
2020-03-30 11:38:44 - All done.