Hot vs bulk: placing app state on a two-tier ZFS homelab

A homelab with a fast single NVMe pool and a slow redundant HDD array has to answer one question per service: does this state go on the fast disk or the safe disk. This is the reference for that decision. It sits beside ZFS on NixOS, which owns the pool and mount mechanics (import, zfsutil, the reexec trap, why snapshots are not backups); this doc owns where the bytes land and why.

TL;DR:

Three pools by hardware role: rpool (single 1TB NVMe, non-redundant, hot state) mounts /appdata and /appdata/pg; tank (6-wide raidz2 HDD, redundant) mounts /tank/media and /tank/appdata; scratch (single 2TB QLC NVMe, sync=disabled) holds transient downloads and transcode temp.
Placement is decided on three axes: latency need, redundancy need, re-derivability. fsync-heavy random-IO state (SQLite, Postgres, redis) earns NVMe; large re-pullable or regenerable data (model weights, object caches, the metrics database) tolerates bulk HDD.
The NVMe tier is a single disk with no pool redundancy. Irreplaceable data still lives there for speed - its protection is ZFS snapshots plus off-pool replication, not a mirror. Irreplaceable does not mean redundant pool; it means snapshotted and replicated.
Placement follows the data’s IO shape, not the service’s label. The Jellyfin media files are bulk on /tank/media, but the Jellyfin library is SQLite hot state on /appdata.
One service’s hot state lives in exactly one path. A prior split scattered state across a /tank/data/appdata cold copy and a /rpool/cache hot copy; it was consolidated on 2026-09-03 into /appdata (hot) and /tank/appdata (bulk), ~40G moved.

The two tiers

Three pools, split by the role the hardware suits. The mechanics - which pool gets imported at boot, who owns each mount, why the non-redundant pool is safe to run without a mirror - are in ZFS on NixOS. What this doc adds is the rule for deciding which service’s state goes where.

Pool	Hardware	Redundancy	Mounts	Allocated
`rpool`	Samsung 970 EVO 1TB, TLC + DRAM, CPU-direct NVMe	None, single disk	`/appdata`, `/appdata/pg`	~158G of ~928G
`tank`	raidz2 HDD, 6-wide (growing to 7-8 via raidz expansion)	2-disk fault tolerance	`/tank/media`, `/tank/appdata`	~32T of ~76T
`scratch`	2TB NVMe, QLC, DRAM-less	None, `sync=disabled`	download staging, transcode temp	transient

tank grows in place: raidz expansion adds disks to an existing raidz2 vdev without recreating the pool.¹

The placement decision

Every piece of state is scored on three axes, and the axis that dominates picks the tier.

Data	Tier	Deciding axis
arr configs (radarr, sonarr, lidarr, prowlarr, bazarr, jellyseerr - SQLite)	`/appdata` NVMe	latency: SQLite journal/WAL fsync
Postgres clusters (immich, atuin, joplin, tracearr timescaledb, mnemosyne, memledger, vaultwarden)	`/appdata/pg` NVMe	latency: random-page fsync
redis / valkey (redis_immich, tracearr-redis, bonkled valkey)	`/appdata` NVMe	latency: append-only fsync
qbittorrent state, beets, atuin	`/appdata` NVMe	latency: frequent small writes
Jellyfin library (SQLite)	`/appdata` NVMe	latency: IO shape, not the service label
media files	`/tank/media` HDD	bulk, write-once (see the migration guide)
MinIO object store	`/tank/appdata` HDD	re-derivability + redundancy over speed
Prometheus TSDB	`/tank/appdata` HDD	re-derivability: a lost window is cheap
LLM model weights (open-webui, gumshoe)	`/tank/appdata` HDD	re-derivability: re-downloadable
research datasets, geo data, searxng config	`/tank/appdata` HDD	re-derivability + redundancy

The three sections below take the axes one at a time.

What earns the NVMe tier

The hot tier is for state whose access pattern is small, random, and synchronous. A relational or embedded database commits by writing to a log and calling fsync, and it does this on every transaction; the cost of that fsync is a function of how long the underlying disk takes to make a durable write at a random offset. On an HDD that is a seek plus a rotation - single-digit to low-tens of milliseconds each - and a database doing hundreds of small commits a second stalls on the mechanics of the platter. On NVMe the same durable write is tens of microseconds. The gap is three orders of magnitude, and it lands entirely on the workloads that commit constantly.

That is why the hot tier holds:

Every arr service config, which is SQLite with its -wal and -shm sidecars. SQLite in WAL mode still fsyncs at each checkpoint, and its write path is the random-small-durable shape HDDs are worst at.²
All Postgres clusters, under /appdata/pg. Postgres writes 8 KB pages and flushes its write-ahead log on commit;³ the same seek-bound penalty applies, multiplied across every service that keeps a Postgres database.
redis / valkey with append-only persistence, which fsyncs its AOF on the configured interval.

Media files are the opposite shape - large, sequential, written once and read streaming - so they sit on the HDD array without penalty. The distinction is not database against files; it is random-synchronous against sequential-streaming. The tier follows the IO shape.

Where redundancy actually comes from

The NVMe tier is a single disk. It has no pool redundancy at all - a dead rpool device loses every byte on it at once. Yet it holds data that is genuinely irreplaceable: the Postgres clusters behind vaultwarden, immich, and the personal tooling. That pairing looks wrong until you separate two things that “redundancy” usually bundles together.

Pool redundancy (a mirror or raidz) protects against one class of failure: a disk dying. It does nothing for the failures that actually destroy homelab data - a bad DELETE, a botched migration, ransomware, an rm -rf on the wrong path. A mirror faithfully replicates the mistake to both disks in real time.

What protects against those is a point-in-time copy you can roll back to, and a copy on different hardware. On this host both come from ZFS: snapshots give the rollback point, and replication of those snapshots to tank (and off-box) gives the second copy. A dead NVMe then costs a reinstall plus the delta since the last replication - a bounded, understood loss - not the data itself.

So irreplaceable data goes on the fast non-redundant tier deliberately. Its safety is snapshots plus replication, and putting it on a mirror instead would buy protection against the least likely failure while costing the second M.2 slot and gating writes. How each tier is snapshotted and replicated is the subject of declarative homelab backups; the point here is only that irreplaceable data routes on latency, and protection is a separate mechanism layered on top.

What belongs on bulk HDD

The bulk tier is for data where losing speed costs nothing you notice and losing the data costs little to recover. Three kinds qualify:

Re-downloadable. LLM model weights (open-webui, gumshoe) are large and pull again from their source; there is no reason to spend NVMe capacity on a file you can fetch. Research datasets are the same shape.
Regenerable or losable-window. The Prometheus TSDB is append-heavy but tolerant: losing the most recent scrape window is a gap in a graph, not a corrupted system, and the data rebuilds as new scrapes land. It reads back rarely and sequentially.
Large object stores and caches. MinIO holds objects that are either re-creatable or already backed elsewhere; the searxng and geo config are small and re-deployable.

For all of these the HDD array’s redundancy is the feature that matters and its latency is the cost that does not. raidz2 rides out a two-disk failure and resilvers; a slow read on a metrics query or a model load is invisible against the workload. The axis that dominates here is re-derivability - cheap to lose, so optimise for the redundant tier - with redundancy as the tie-breaker.

The Jellyfin library edge case

Jellyfin is the case that shows why the tier follows the data and not the service. Jellyfin is “media”, and its media files are exactly what the bulk tier is for: large, write-once, streamed. They live on /tank/media.

Its library is a different animal. The Jellyfin library is a SQLite database - metadata, watch state, image references - with the same random-synchronous write pattern as any other SQLite config. Placed on the HDD array it would inherit the seek-bound fsync penalty on every scan and playback update. So the library lives on /appdata, the hot tier, next to the arr configs, while the files it indexes stay on bulk.

One service, two placements, because it carries two kinds of state. A rule that placed data by service label (“Jellyfin is media, put it on the media pool”) would put the database on the wrong tier. The rule that works is: score the state, not the service.

Dataset boundaries and recordsize

Each database keeps its files whole on one dataset, so a snapshot of that dataset is crash-consistent for that database - the reason ZFS on NixOS gives for not splitting a write-ahead log onto a separate dataset from its main file.

The Postgres dataset /appdata/pg carries recordsize=16K. Postgres reads and writes in 8 KB pages;³ a 16K ZFS record covers two pages, which is a common default for Postgres on ZFS. The reasoning is a balance rather than a measurement here: a recordsize at or near the page size keeps a single-page write from forcing a read-modify-write of a much larger record (write amplification), while staying large enough that lz4 still finds something to compress. The 16K value is what the dataset is set to; the balance argument is the rationale for it, not a benchmark run on this host.

The /appdata datasets otherwise take the pool defaults (lz4, atime=off), matching the property table in ZFS on NixOS.

The one-canonical-path rule

Before the 2026-09-03 consolidation, app state was split by accident: some services wrote a hot copy to /rpool/cache while a cold copy of the same state sat under /tank/data/appdata. Two locations for one service’s state means two things to back up, two things to reason about when restoring, and a standing question of which copy is authoritative.

The consolidation collapsed that into exactly two canonical locations, /appdata for hot and /tank/appdata for bulk, moving ~40G. The move itself ran through migctl - cross-dataset copy, checksum re-verify, coverage diff, and a gate that does not clear until the copy is provably complete - covered in Migrating a NAS without losing a byte. The rule it enforces: one service’s hot state lives in exactly one path. appdata is appdata. A service is either hot or bulk, and its state is in the one directory that says so - which is also what makes the snapshot policy checkable, because there is a single dataset per tier to verify coverage against.

Decision guide

Reading the guide as a list:

If the state is small, random, and synchronous - any SQLite, Postgres, or redis workload, including a media server’s library - it goes on the hot NVMe tier /appdata. Its safety comes from snapshots and replication, not from a redundant pool.
Otherwise, if it is cheap to lose or re-derive - re-downloadable weights, regenerable metrics, re-creatable object caches - it goes on the bulk HDD tier /tank/appdata, where redundancy matters and latency does not.
Otherwise, if it is large, sequential, and written once - the media files themselves - it goes on /tank/media.

Score the state, not the service, and give each service exactly one hot path and one bulk path.

ZFS on NixOS - who owns the mount, how the pools import, the reexec trap, and why the non-redundant pool is safe to run. The mechanics under this doc’s placement decisions.
Migrating a NAS without losing a byte - how the consolidation move was executed and verified, with a gate that does not clear until the copy is provably complete.
Declarative homelab backups - how each tier is protected: the snapshot policy and off-pool replication that stand in for the NVMe tier’s missing redundancy.

References

OpenZFS, “zpool-attach.8,” OpenZFS Documentation. https://openzfs.github.io/openzfs-docs/man/master/8/zpool-attach.8.html ↩
SQLite, “Write-Ahead Logging,” SQLite Documentation. https://www.sqlite.org/wal.html ↩
PostgreSQL, “Database Page Layout,” PostgreSQL Documentation. https://www.postgresql.org/docs/current/storage-page-layout.html ↩ ↩²