Backup the Operational Database

Scouter Analytics manages operations with SQLite, and for that reason you're free to use any of the multitude of SQLite backup options. This guide covers litestream because its continuous streaming replications gives us point-of-failure recovery.

First, you must install litestream.

Litestream can be configured to write to a multitide of different backends. This guide will cover S3 compatible object storage.

You'll need to generate a key for your S3 compatible API of choice and set the correct bucket, path, and endpoint.

/etc/scouter/litestream.yml

access-key-id: 
secret-access-key: 
dbs:
  - path: ${STATE_DIRECTORY}/scouter.db
replicas:
  - type: s3
    path: 
    bucket: 
    endpoint: 
    sync-interval: 10s
    logging:
      level: info

/etc/systemd/system/analytics-backup.service

[Unit]
Description=Scouter Analytics Backups
PartOf=analytics.target

[Service]
Restart=on-failure
DynamicUser=yes
ProtectSystem=full
NoNewPriviliges=true
PrivateDevices=true
StateDirectory=scouter
ExecStart=/usr/local/bin/litestream replicate -config /etc/scouter/litestream.yml

[Install]
WantedBy=multi-user.target

There's a restore unit to automatically restore the database from the latest backup. This is useful to 1) ensure the service can start without manual intervention, 2) you can make the StateDirectory ephemeral and have it wiped every time the service restarts.

/etc/systemd/system/analytics-restore.service

[Unit]
Description=Scouter Anaytics Restore
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
StateDirectory=scouter
Delegate=yes
DynamicUser=yes
ProtectSystem=full
NoNewPrivileges=true
PrivateDevices=true
NotifyAccess=all
ExecStart=/usr/local/bin/litestream \
  restore \
  -config=/etc/scouter/litestream.yml \
  -if-db-not-exists \
  -if-replica-exists ${STATE_DIRECTORY}/scouter.db

Configure Scouter Analytics to only start after the database has been restored, and the backup service is running.

/etc/systemd/system/analytics.service.d/backup-and-restore.conf

[Unit]
Requires=analytics-restore.service analytics-backup.service
After=analytics-restore.service