Declarative backups on a ZFS homelab: nix timers, sanoid, syncoid

Backups on this NAS used to run as GitOps pipelines that exec-ed shell scripts into idle containers. They were replaced by one NixOS module that owns three mechanisms and a matrix saying which one protects which data. The change gives coverage you can derive from the configuration, loaded at nixos-rebuild switch rather than drifting inside a container. This page covers that module and the two silent failures that retired the imperative version.

TL;DR:

Three mechanisms, split by data shape. Postgres clusters get logical pg_dump timers; file state gets sanoid snapshots plus a syncoid replica. No separate tar of the files - the snapshot and the replica already carry them.
A snapshot on the same pool is not a backup. syncoid copies rpool/appdata to the redundant tank, which is the step that turns a point-in-time snapshot into an off-pool copy.
The imperative version broke twice in ways a declarative one cannot: a backup that read a dead path and kept reporting success, and the general class of a script that is written, tested, committed, and never installed.
All Postgres clusters are treated as tier-1. Baseline is an hourly snapshot plus a daily logical dump; WAL PITR is the plan for the clusters whose data is not regenerable.
One cluster’s dump is file-verified but restore-unverified. A green timer does not show that a dump restores; only a test restore does.

Topology

The same source tree feeds all three mechanisms:

Each Postgres cluster on /appdata/pg is dumped by its own timer into /tank/backups/pg, and the timer emits a metric on success.
sanoid snapshots both /appdata and /appdata/pg in place on rpool.
syncoid ships those existing snapshots to tank/backups/zfs-repl on the redundant pool.

Which mechanism protects what

Data	Lives on	Mechanism	Form	Off-pool copy
5 Postgres clusters (immich, atuin, joplin, tracearr timescaledb, mnemosyne)	`rpool/appdata/pg`	`pg_dump` timer, one per cluster	logical dump (`pg_dump -Fc`)	`tank/backups/pg`
Arr SQLite configs, redis, qbittorrent, beets, atuin hot state	`rpool/appdata`	`sanoid` snapshot + `syncoid` replica	ZFS block state	`tank/backups/zfs-repl`

Given a piece of data, the matrix names the mechanism that protects it and the redundant pool that holds the copy, without reading a script. File state needs no separate tar, because the snapshot and the replica capture the arr SQLite files, the redis and qbittorrent state, the beets library, exactly as they sit on disk. This split is stated verbatim in backup.nix.

What makes a copy a backup

A snapshot on the pool it protects dies with the pool; ZFS on NixOS covers that, and the pool and mount mechanics. A backup needs two properties a snapshot alone lacks:

It is off-pool: the copy lives on different, redundant media, so losing the source pool does not lose the copy. syncoid provides this by replicating to tank.
It is restore-tested. A copy that has never been restored may not restore; the untested-restore trap below is the case on this host.

The retired cron pipelines had the first property and not the second. They wrote files to tank, so the copies were off-pool, but nothing checked that a file held the current data or that anyone had restored it, and the imperative shape hid both gaps.

The three mechanisms

1. pg_dump timers - logical dumps, one per cluster

Each of the five Postgres clusters (immich, atuin, joplin, the tracearr timescaledb, and mnemosyne) has its own systemd timer. The runner does docker exec into each cluster and reads that container’s own environment for credentials (PGPASSWORD, POSTGRES_USER, POSTGRES_DB), so the dump uses the same secret the database already holds and the module carries no copy of it. It runs pg_dump -Fc --no-owner --no-acl,¹ writes to /tank/backups/pg/<db>, keeps a retention window, and emits two node-exporter textfile metrics on success: backup_pg_last_success_timestamp and backup_pg_last_bytes.²

The schedule is OnCalendar=*-*-* 02:30:00 with a randomised delay, so the five timers do not all fire on the same second.³ The custom format (-Fc) is a logical dump: it restores into any Postgres with pg_restore, and it survives a major-version bump, which a raw file-level copy of the data directory does not.¹

2. sanoid snapshots - point-in-time on the hot tree

sanoid runs one template, apphot, on rpool/appdata and rpool/appdata/pg:⁴

Cadence	Kept
hourly	24
daily	7
weekly	4
monthly	3

Snapshots are point-in-time and near-free on a mostly-idle NVMe tree - a snapshot records only the blocks that later diverge, so an idle dataset costs almost nothing to keep 38 of. The retention section gives the reason for all four cadences.

3. syncoid replication - the step that makes it a backup

syncoid replicates rpool/appdata to tank/backups/zfs-repl, recursively, so the pg child dataset comes along.⁴ It runs with --no-sync-snap, which ships the snapshots sanoid already cut rather than taking its own,⁵ so both tools work from one set of snapshots.

Everything sanoid does stays on rpool; only after syncoid runs does a copy exist that survives losing that NVMe.

Why the pipelines were retired

Both of these failure modes happened before the imperative version was retired.

The dead-path backup

The old appdata_backup carrier bind-mounted /rpool/cache/data as its source. A storage move later relocated that state to /appdata and quarantined /rpool/cache. The carrier kept exec-ing its tar on a schedule and exiting zero, so it reported success while reading a path that no longer held any live data. The backup was stale and effectively empty, and nothing flagged it.

The written-but-never-installed class

An imperative backup script can be authored, unit-tested, committed, and documented, and still never run, because installing and scheduling it is a separate manual step that nobody did. Nothing enforces that the script on disk is a job in the scheduler.

A NixOS module closes that gap by construction: the timer and its service are declared in the same config that gets applied, so a timer that evaluates at switch is a timer that exists on the system. There is no second step in which the intent can be lost.

RPO tiers: baseline vs WAL PITR

Tier	Clusters	Baseline RPO	Target RPO
Tier-1, cannot lose	all Postgres clusters, with vaultwarden and atuin called out (their data is not regenerable)	hourly snapshot + daily logical dump	as baseline, until WAL PITR lands
Tier-1, highest value	vaultwarden, memledger, atuin	as above	seconds, via WAL PITR

Every Postgres cluster is tier-1: the decision was that none of them may lose data, so the baseline - an hourly snapshot plus a daily logical dump - applies to all of them. The worst-case loss under the baseline is roughly the snapshot interval, about an hour. vaultwarden and memledger are the two Postgres clusters outside the five pg_dump timers: their own compose stacks run dump sidecars, which backup.nix deliberately leaves in place.

On top of that, continuous WAL archiving (point-in-time recovery) is the plan for the clusters whose data cannot be regenerated at all - vaultwarden, memledger, atuin. WAL PITR ships every committed transaction to the archive as it happens, which shrinks the recovery point from about an hour to seconds.⁶ This is design only: the baseline runs today, and the WAL archiving does not yet.

Retention: why hot state needs all four cadences

The apphot template keeps hourly, daily, weekly, and monthly snapshots because hot state has two independent recovery needs, and neither cadence covers both:

A short loss window. If a database corrupts a row or a service writes garbage, the useful snapshot is minutes old. The 24 hourly snapshots give a recovery point within about an hour across a full day.
A long tail. Some damage is noticed late: a slow data-quality regression, or a config change whose effect surfaces weeks on. The weekly and monthly snapshots hold a restore point going back roughly three months without keeping an hourly for every one of those hours.

Drop the hourly cadence and a mid-day mistake costs up to a day; drop the weekly and monthly, and anything noticed after a week has no restore point. The template therefore carries the full ladder instead of the two-cadence hot template that ZFS on NixOS described as a temporary measure.

Alerting on failure and on silence

The imperative version failed silently by default. The declarative version wires two controls so a failure surfaces:

OnFailure=backup-alert@%n on every timer’s service. If any run exits non-zero, systemd starts the templated alert unit with the failed unit name as its instance, so a failure raises a notification instead of a log line nobody reads.⁷
A stale-metric alert. The pg_dump runner writes backup_pg_last_success_timestamp to the node-exporter textfile collector,² which the monitoring stack scrapes. An alert on that timestamp going stale catches a timer that never fired at all, which the OnFailure hook cannot see because nothing failed. Where those metrics land and how the scrape reaches this host is covered in self-hosted monitoring topology.

The untested-restore trap

A dump that exists on disk is not a backup you can trust until a restore has been exercised against it. The backup_pg_last_bytes metric proves a file of non-zero size was written; it proves nothing about whether that file restores into a working database.

On this host, mnemosyne was parked mid-migration. Its data was copied to the new tier, and the file is present and non-empty, but Postgres has never opened it there, so the copy is file-verified and restore-unverified. Until a pg_restore into a scratch instance comes up clean, that cluster has a file that is probably a backup. A green timer and a healthy-looking byte count do not change that.

Decision guide

The same tree as a list, with the reason for each branch:

File state (arr SQLite, redis, qbittorrent, beets) -> sanoid snapshot plus syncoid replica, because block-level snapshots capture the files crash-consistently and a separate tar would duplicate what the replica already carries.
A Postgres cluster whose data is regenerable -> a logical pg_dump timer plus the hourly snapshot baseline, because a logical dump restores across major versions and the hourly snapshot bounds the loss window to about an hour.
A Postgres cluster whose data cannot be regenerated (vaultwarden, memledger, atuin) -> everything in row 2 plus continuous WAL archiving, because only per-transaction archiving shrinks the recovery point to seconds.

ZFS on NixOS: who owns the mount - the snapshots-are-not-backups principle in full, plus the pool and mount mechanics this doc rests on.
Monitoring a self-hosted stack behind a policy-drop edge - where the backup_pg_last_success_timestamp metrics land and how the scrape reaches this host.
Appdata tiering on ZFS - which tier holds what you are protecting, and why hot state sits on the non-redundant NVMe.

References

PostgreSQL, “pg_dump,” PostgreSQL Documentation. https://www.postgresql.org/docs/current/app-pgdump.html ↩ ↩²
Prometheus, “node_exporter textfile collector,” GitHub. https://github.com/prometheus/node_exporter#textfile-collector ↩ ↩²
freedesktop.org, “systemd.timer,” systemd Manual. https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html ↩
Jim Salter, “sanoid and syncoid,” GitHub. https://github.com/jimsalterjrs/sanoid ↩ ↩²
Jim Salter, “syncoid,” sanoid README. https://github.com/jimsalterjrs/sanoid/blob/master/README.md ↩
PostgreSQL, “Continuous Archiving and Point-in-Time Recovery,” PostgreSQL Documentation. https://www.postgresql.org/docs/current/continuous-archiving.html ↩
freedesktop.org, “systemd.unit,” systemd Manual. https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html ↩