clewsy
A work-in-progress
ncbu - or NextCloud Back-Up - is a docker container image created to simplify and automate perodic physical snapshots of a self-hosted nextcloud service. The snapshots serve as a backup of the nextcloud volume and also (if applicable) the nextcloud database volume.
The intention is to run this image as a container alongside the nextcloud app and database containers. Management of the "live" nextcloud/database volumes can therefore be left to docker. When the ncbu back-up script is initialised, it first locks the nextcloud instance (i.e. puts nextcloud into maintenance mode) before syncing the volumes to a user-defined location. Once complete, the script unlocks nextcloud so that normal usage can resume.
By locking nextcloud first, ncbu ensures that the backup is effectively a "snapshot" of the nextcloud and database volumes.
I discovered docker on my journey to rid myself of cloud-based services not within my control. A docker implementation of nextcloud is great for robustness and portability. It also facillitated simple back-ups by binding the data volumes to a user-accessible directory. Initially I set up an rsync cronjob to automate my backups and this was fine, but I saw potential benefits in containerising the backup service. Mostly, I wanted to have a go at learning docker and creating my own container image.
Now my backup automation is implmented simultaneously alongside the nextcloud and database containers thanks to docker-compose and a single *.yml file. Refer to my clews.pro project for more information of my self-hosting implementation.
Once spun up, provided the correct environment variables and volumes are defined, ncbu will run an initialisation script (ncbu_init) to do some basic error-checking, then it will simply kick off cron in the foreground. There is a single cron job set (at a user-defined time, midnight every day by default) which will execute the backup script.
The backup script (ncbu) will first put the nextcloud instance into maintenance mode, effectively "freezing" the nextcloud data files and database. Rsync is then used to copy all of the files from the nextcloud and database volumes to a backup directory. Once complete, maintenance mode is disabled.
A third script within the container (ncbu_restore) is used to restore nextcloud and database containers from a stored backup. The restore script can only be run manually by the user. It will similarly put nextcloud into maintenance mode and then sync all the nextcloud and database files in the reverse direction before exiting maintenance mode. The restore script will also initialise a scan of the data files and update the database accordingly - this helps if the user is "restoring" to a new host.
The following commands may be useful, but much more detail can be found at the gitlab or dockerhub repositories.
Build from source:
$ git clone git@gitlab.com:clewsy/ncbu
$ cd ncbu
$ docker build -t ncbu/ncbu .
Pull from dockerhub:
$ docker pull clewsy/ncbu
Run using the docker run command:
$ docker run -ti \
-h ncbu \
-e NEXTCLOUD_CONTAINER=nextcloud \
-e NEXTCLOUD_DATABASE_CONTAINER=nextcloud-db \
-e NEXTCLOUD_BACKUP_CRON="0 0 * * *" \
-v /etc/localtime:/etc/localtime:ro \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v nextcloud-app:/mnt/nextcloud-app \
-v nextcloud-db:/mnt/nextcloud-db \
-v ./nextcloud-bu:/backup \
clewsy/ncbu
Alternatively, a better/recommended method of running ncbu is via docker-compose. Here is a link to an example docker-compose.yml file.
$ cd /home/docker #Substitute directory for location of the docker-compose.yml file.
$ docker-compose pull && docker-compose up -d
In any case, if configured correctly, the container should be running. This can be confirmed by running the command docker ps (or docker-compose ps if using docker-compose). The output of this command will show the status of all running containers.
$ cd /home/docker #Substitute directory for location of the docker-compose.yml file.
$ docker-compose ps
| Name | Command | State | Ports |
| ----------------- | --------------------------------- | --------------- | ---------------------------------------- |
| collabora | /bin/sh -c bash start-libr ... | Up | 9980/tcp |
| letsencrypt | /bin/bash /app/entrypoint. ... | Up | |
| nextcloud | /entrypoint.sh apache2-for ... | Up | 80/tcp |
| nextcloud-bu | ncbu_init | Up (healthy) | |
| nextcloud-cron | tini -- /entrypoint.sh | Up (healthy) | |
| nextcloud-db | /init | Up | 3306/tcp |
| nginx-proxy | /app/docker-entrypoint.sh ... | Up | 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp |
I included a "healthcheck" script (ncbu_healthcheck) in the container image which is executed every 10 minutes. As per the example above, the container state should read "Up (healthy)" if everything is running as expected.
The script will identify the state as "unhealthy" if either of two conditions are met:
Confirm inintialisation was successful by checking the log. This should clarify any errors that may have occurred. Command below followed by output for a successful initialisation:
$ docker logs ncbu
2020-03-30 11:13:29 - Initialising ncbu (nextcloud-backup)...
2020-03-30 11:13:29 - Checking environment variables provided:
NEXTCLOUD_CONTAINER=nextcloud
NEXTCLOUD_DATABASE_CONTAINER=nextcloud-db
2020-03-30 11:13:29 - Checking mounted volumes:
Nextcloud app volume successfully mounted to /mnt/nextcloud_app
Nextcloud database volume successfully mounted to /mnt/nextcloud_db
2020-03-30 11:13:29 - Checking backup volume:
Backup exists.
2020-03-30 11:13:29 - Updating crontab with: "0 0 * * * ncbu"
2020-03-30 11:13:29 - Running crond in the foreground...
$ docker exec ncbu ncbu
2020-03-30 10:22:14 - Running ncbu (nextcloud backup)...
2020-03-30 10:22:14 - Putting nextcloud into maintenance mode...
Maintenance mode enabled
2020-03-30 10:22:14 - Nextcloud data backup: Syncing nextcloud volume to /backup...
sending incremental file list
config/config.php
1.02K 100% 0.00kB/s 0:00:00 (xfr#1, ir-chk=1026/15502)
sent 1.12M bytes received 4.14K bytes 749.39K bytes/sec
total size is 24.82G speedup is 22,078.86
2020-03-30 10:22:15 - Finished nextcloud data sync.
2020-03-30 10:22:15 - Setting permission of nextcloud data directory to :33
2020-03-30 10:22:15 - Ensure user on host machine is part of group id GID=33 for read access to backup.
2020-03-30 10:22:15 - Nextcloud database backup (physical copy): Syncing nextcloud-db volume to /backup...
sending incremental file list
sent 9.77K bytes received 18 bytes 19.58K bytes/sec
total size is 691.70M speedup is 70,661.44
2020-03-30 10:22:15 - Finished nextcloud database sync.
2020-03-30 10:22:15 - Taking nextcloud out of maintenance mode...
Maintenance mode disabled
2020-03-30 10:22:15 - All done.
$ docker exec ncbu ncbu_restore
2020-03-30 11:34:12 - Running ncbu restore...
2020-03-30 11:34:12 - Putting nextcloud into maintenance mode...
Maintenance mode enabled
2020-03-30 11:34:12 - Nextcloud data restore from ncbu: Syncing /backup/nextcloud_app to nextcloud-app volume...
sending incremental file list
config/config.php
1.02K 100% 0.00kB/s 0:00:00 (xfr#1, ir-chk=1026/15502)
sent 1.12M bytes received 4.14K bytes 749.46K bytes/sec
total size is 24.82G speedup is 22,079.13
2020-03-30 11:34:13 - Finished nextcloud data sync from backup.
2020-03-30 11:34:13 - Nextcloud database restore from ncbu physical copy: Syncing /backup/nextcloud_db to nextcloud-db volume...
sending incremental file list
sent 9.82K bytes received 18 bytes 19.67K bytes/sec
total size is 692.82M speedup is 70,429.63
2020-03-30 11:34:13 - Finished nextcloud database sync from backup.
2020-03-30 11:34:13 - Taking nextcloud out of maintenance mode...
Maintenance mode disabled
2020-03-30 11:34:14 - Scanning the nextcloud data files and updating the cache accordingly.
(This may take a while but the nextcloud instance should be accessible while the scan runs.)
Starting scan for user 1 out of 1 (jadon)
Folder /jadon/
Folder /jadon/files_trashbin
Folder /jadon/files_trashbin/files
File /jadon/files_trashbin/files/Michael Crichton - Timeline.epub.d1581224543
File /jadon/files_trashbin/files/Nextcloud intro.mp4.d1580532366
File /jadon/files_trashbin/files/wami_irl_b4t-phone_daily_2020-02-28_b4t-phone.gpx.d1584916818
File /jadon/files_trashbin/files/wami_irl_monthly_2019-12_b4t-phone.gpx.d1584916819
File /jadon/files_trashbin/files/wami_irl_b4t-phone.gpx.d1580552369
File /jadon/files_trashbin/files/Screenshot from 2020-01-27 11-57-03.png.d1581051578
File /jadon/files_trashbin/files/wami_irl_b4t-phone_daily_2020-02-14_b4t-phone.gpx.d1584916817
File /jadon/files_trashbin/files/Screenshot from 2020-01-27 11-54-06.png.d1581051577
File /jadon/files_trashbin/files/Screenshot from 2020-01-27 11-53-25.png.d1581051587
Folder /jadon/files_versions/InstantUpload/Camera/2020/02
File /jadon/files_versions/InstantUpload/Camera/2020/02/IMG_20200202_165550.jpg.v1580622950
Folder /jadon/files_versions/Photos
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 248 | 17352 | 00:03:04 |
+---------+-------+--------------+
2020-03-30 11:38:44 - All done.
To assist in tracking down any issues, the scripts output info to a logfile. All output from any rsync commands also goes into the logs.
The logfile is located within the /backup directory within the container. Provided this directory is bound to another directory oin the host machine, then it can be read by the host. If the previously referenced example docker-compose.ymlis used along with the demo configurations above, then the logfile will be located at /home/docker/nextcloud-bu/ncbu.log. Example logs are shown below:
$ cat /home/docker/nextcloud-bu/ncbu.log
2021-06-18 00:00:00 - Running ncbu (nextcloud backup)...
2021-06-18 00:00:00 - Putting nextcloud into maintenance mode...
2021-06-18 00:00:00 - Nextcloud data backup: Syncing nextcloud-app volume to /backup...
2021/06/18 00:00:00 [35874] building file list
2021/06/18 00:00:01 [35874] >f..t...... config/config.php
2021/06/18 00:00:02 [35874] >f.st...... data/nextcloud.log
2021/06/18 00:00:02 [35874] >f.st...... data/appdata_oc037zsrujze/appstore/apps.json
2021/06/18 00:00:02 [35874] .d..t...... data/appdata_oc037zsrujze/preview/0/6/c/d/4/1/6/
2021/06/18 00:00:06 [35874] sent 67.01M bytes received 44.01K bytes 10.32M bytes/sec
2021/06/18 00:00:06 [35874] total size is 34.27G speedup is 511.07
2021-06-18 00:00:06 - Finished nextcloud data sync.
2021-06-18 00:00:06 - Setting permission of nextcloud data directory to :33
2021-06-18 00:00:07 - Ensure user on host machine is part of group id GID=33 for read access to backup.
2021-06-18 00:00:07 - Nextcloud database backup (physical copy): Syncing nextcloud-db volume to /backup...
2021/06/18 00:00:07 [35901] building file list
2021/06/18 00:00:07 [35901] >f..t...... databases/ib_logfile0
2021/06/18 00:00:08 [35901] >f..t...... databases/ib_logfile1
2021/06/18 00:00:10 [35901] >f.st...... log/mysql/mariadb-bin.index
2021/06/18 00:00:10 [35901] sent 386.56M bytes received 436 bytes 110.45M bytes/sec
2021/06/18 00:00:10 [35901] total size is 878.59M speedup is 2.27
2021-06-18 00:00:10 - Finished nextcloud database sync.
2021-06-18 00:00:10 - Taking nextcloud out of maintenance mode...
2021-06-18 00:00:10 - Rotating logfile if required...
2021-06-18 00:00:10 - All done.
