Forgejo as the primary forge on the NixOS edge router

git.erfi.io is the git and CI primary for the whole personal fleet. A single-user Forgejo instance on the NixOS edge router serves every repository and runs every workflow; GitHub holds a push-mirror backup of the git data with Actions switched off. This page covers the topology, the split between router disk and object store, edge TLS and SSH, hardening, backups and the restore drill, and when GitHub-primary is still the better pick.

Measured rows come from a read-only probe run on 2026-09-29 around 06:20 +08 against the router (docker inspect, select count(*) in Postgres, greps of named keys, systemctl show) and GET requests to the public API and the Composer API. Configuration rows come from the stack’s compose files, the router’s NixOS config, and Forgejo’s app.ini.

TL;DR

The Forgejo-primary migration ran 2026-09-23 to 2026-09-27. Live inventory on 2026-09-29: 269 repositories, 0 of them pull mirrors, 0 public; 206 push mirrors to GitHub, one per mirrored repo, all syncing on commit and on an 8h interval.
Git repositories and Postgres live on router Docker volumes. Everything in Forgejo’s [storage] section lives in one bucket on Silo, an S3-compatible fork of MinIO on the storage host, reachable through exactly one nftables forward rule.
Hardening: rootless image, cap_drop: ALL, no-new-privileges, memory limits of 2048M (Forgejo), 1024M (Postgres), 2048M (runner), CPU limit 4 each.
DEFAULT_ACTIONS_URL is set to github; Forgejo’s own default is https://data.forgejo.org.¹
A systemd timer at 02:30 dumps Postgres with pg_dump -Fc and rsyncs the bare-repo tree to the storage host’s ZFS backup dataset. The restore drill passed 2026-09-23 (recorded in the migration plan, not re-run for this page).

Topology

The same thing in list form:

Git clients and the browser reach the edge Caddy on 443 (HTTPS) and 2223 (SSH).
Caddy proxies to Forgejo on the pinned bridge; SSH is an L4 proxy to Forgejo’s built-in SSH server.
Forgejo keeps the git repositories and Postgres on router volumes.
Everything in Forgejo’s [storage] section goes to Silo, the S3-compatible object store on the storage host, over one forward-chain rule.
Every mirrored repo push-mirrors to GitHub.
A nightly timer dumps Postgres and rsyncs the repositories to the storage host’s ZFS backup dataset.
Composer redeploys stacks on webhook.

Which do I pick

The migration is a decision about where CI runs and which copy is canonical, with a backup direction to match.

Question	Forgejo primary + GitHub push mirror (this setup)	GitHub primary + Forgejo pull mirror (the pre-2026-09-23 setup)
Where CI runs	Self-hosted runner on your own hardware	GitHub Actions; Forgejo pull-mirror repos never get an Actions unit, so a self-hosted runner cannot run their workflows
What GitHub holds	The git data (branches, tags, commits), pushed on every commit and every 8h²	The canonical copy, including issues, PRs, releases
Issues, PRs, release metadata	Start empty on Forgejo; nothing is copied from GitHub	Stay on GitHub
Outage of your box	Git and CI stop; the GitHub backup keeps a git copy	Development continues on GitHub
Deploy tooling bootstrap	The forge’s own stack must not be sourced from the forge	No circularity
Credential to rotate	One GitHub token stored in every push-mirror row	Tokens in the pull-mirror remotes
Backups you own	Postgres dump + repo mirror + Silo snapshots	GitHub is the backup of record

Component versions

Component	Version	Notes
Forgejo	`codeberg.org/forgejo/forgejo:16.0.5-rootless`; public API reports `16.0.5+gitea-1.22.0`	Single user; self-registration disabled
PostgreSQL	`postgres:18-alpine`; live server reports 18.6	Upgraded from 17 to 18 on 2026-08-29 onto a new volume; the old pg17 volume is still declared for rollback
Forgejo Runner	`data.forgejo.org/forgejo/runner:13.2.0`	Capacity 4; labels `ubuntu-latest` and `ubuntu-22.04`, both mapped to `ghcr.io/catthehacker/ubuntu:act-22.04`

Why the router

Gitea 1.27.2-rootless ran on the old Unraid NAS. On 2026-08-23 the forge moved to Forgejo on the NixOS edge router, the same host as the edge Caddy proxy, in the first router compose commit of that day. The Gitea compose file is kept unchanged as a rollback variant.

On 2026-09-18 the router became the permanent home; the earlier “until the new server is built” framing was dropped. The stated reason: the router’s CPU is the strongest in the lab, which suits a CI runner. The router runs a 13th Gen Intel Core i9-13900H, 20 logical CPUs, ~31 GiB RAM; the root disk is 937G with 97G used (11%).

Storage split

Since 2026-09-25 the stack splits storage by location:

Router Docker volumes - the git repositories (6.3G on 2026-09-29) and Postgres data.
Silo, an S3-compatible fork of MinIO, on the storage host - everything in Forgejo’s [storage] section: release assets, Actions logs and artifacts, packages, LFS, avatars, attachments, one bucket, each subsystem in its own directory.³ With STORAGE_TYPE = minio, the [storage] section sets the default for all subsystems to the S3-compatible server.³

Reachability is one nftables forward-chain rule: from the forge bridge to the storage host’s S3 port on the servers segment. The forward chain is policy-drop, so without the rule the traffic is dropped. Silo’s data is covered by daily sanoid snapshots of the storage host’s appdata dataset, same pool as the live data with no off-pool copy, which is accepted. Pre-switch local copies of attachments, logs and packages remain on the router volume as frozen leftovers; Forgejo reads from Silo.

Where these datasets sit on the storage host is covered in Hot vs bulk: placing app state on a two-tier ZFS homelab.

Edge: TLS, WAF, SSH

TLS - Caddy’s ACME issuer uses DNS-01 over RFC 2136 (TSIG-signed updates) against the self-hosted Knot authority; the authority itself is covered in knotea: one binary that is both your recursive resolver and your authoritative DNS.
WAF - the site imports an empty placeholder snippet, so git.erfi.io gets no WAF processing; the L4 DDoS jail listener wrapper still applies to every HTTPS connection before the TLS handshake.
Long-lived streams - the reverse proxy disables the response-header timeout and sets flush_interval -1, so long git HTTP, LFS, and Actions log streams are not cut or buffered.
SSH - Caddy’s layer4 app listens on 2223 and proxies to Forgejo’s built-in SSH server on 2222. Forgejo shows 2223 in clone URLs (SSH_PORT) and binds 2222 (SSH_LISTEN_PORT). The router firewall accepts 2223 from WAN and LAN.

Since 2026-09-19 the edge proxy no longer runs as containers. The edge Caddy and its control plane (edgectl, formerly wafctl) are native NixOS services on the router: one caddy-edge binary - Caddy 2.11.4 plus the nine plugins the old container image carried - built by a custom derivation in the router repo (nixpkgs’ withPlugins cannot fetch the private plugin repos; see Edge Caddy as a native NixOS service) and run by the stock services.caddy module, with the Caddyfile and error pages now living in the router repo. Adding an app at the edge is a Caddyfile edit, a knotctl DNS record, and make deploy; the compose stack, its dedicated bridge, and the admin-port hop are gone.

Hardening

Forgejo runs from the rootless image as UID 1000:1000 with cap_drop: ALL and no-new-privileges. Postgres drops all capabilities and adds back only CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID; a live docker inspect shows capdrop=[ALL] on both containers.

Memory limits are 2048M (Forgejo), 1024M (Postgres), 2048M (runner); CPU limit 4 each, and the live cgroup limits match. Forgejo’s /tmp is a 512M tmpfs, which counts against the same memory cgroup as the process.

The 512M tmpfs is why the OOM loop on 2026-09-26 happened: anon-rss reached ~1044MB against a 1GiB limit while the host had ~22GB free, the container restarted 30 times, and the limit went from 1024M to 2048M (also in the rollback file). On 2026-09-29 the container had RestartCount=0 and was up 46h.

The runner is the part of the setup with the least separation. Jobs get the host Docker socket (container.docker_host: automount mounts the host socket into job containers)⁴ and run --privileged (container.privileged: true launches job containers with --privileged)⁵, so workflow code is effectively root on the router. Forgejo’s own framing: the runner performs remote code execution, which poses significant security threats for the host and network it operates on.⁶ That access is what lets a job run its own dockerd for testcontainers-style tests - bind mounts, published ports - instead of a sandboxed one. The cost is that only trusted code can run here: the instance is single-user with registration off, and fork PRs or third-party repos must not run on it.

Incidents that shaped this setup

Date	What broke	What changed
2026-08-23	Job containers on act’s per-workflow bridge could not reach the forge (Docker inter-bridge isolation); `git fetch` timed out	Runner moved to the forge network, plus an `--add-host` fallback, both applied by the `runner_seed` init container
2026-09-21	Runner 13.0.0 against server 16.0.5 fetched tasks, then the job worker vanished silently	Runner upgraded to 13.2.0; keep server and runner patch levels close
2026-09-23	Testcontainers in CI: dials from job containers to published ports were dropped, and an INPUT-chain rule had zero effect because Docker hairpin-DNATs the dial into FORWARD	Forward-chain rule scoped to `ct status dnat` between the Docker bridges
2026-09-24	Runner down ~2h20m: a rotated runner token was not a Forgejo-issued runner token	Runner created in the admin UI; identity declared in config, token written to a 0600 file at start
2026-09-24/25	Docker’s default 10s stop SIGKILLed the runner mid-job and orphaned job containers	`stop_grace_period: 15m` plus an hourly reaper for orphaned job containers
2026-09-26	memcg OOM loop on the Forgejo container, described above	Limit raised 1024M to 2048M

app.ini choices

Key	Value	Why
`DISABLE_REGISTRATION`	`true`	Single-user instance; only an admin can create accounts¹
`DEFAULT_ACTIONS_URL`	`github`	Forgejo’s own default is `https://data.forgejo.org`¹; this instance points it at GitHub instead
`ARTIFACT_RETENTION_DAYS`	`90`	Matches Forgejo’s default; individual uploads can set their own `retention-days`¹
`[webhook] ALLOWED_HOST_LIST`	`private`	Webhooks may only call RFC 1918 / RFC 4193 / RFC 6598 addresses¹; that is what lets them reach Composer on the router’s internal address
`ENABLE_SWAGGER`	`false`	Removes the OpenAPI surface (Forgejo’s default is `true`)¹
`ENABLE_BASIC_AUTHENTICATION`	`false`	Removes basic-auth clone from the edge
`DISABLE_GIT_HOOKS`	`true`	No user-supplied hooks run on the server
`DISABLE_QUERY_AUTH_TOKEN`	`true`	Credentials out of query strings
`OFFLINE_MODE`	`true`	Disables use of CDNs for static files and Gravatar¹
`DISABLE_GRAVATAR`	`true`	No external lookups
`REVERSE_PROXY_TRUSTED_PROXIES`	The forge bridge subnet	Only the pinned bridge is a trusted proxy
`[mail]`	SMTPS port 465 via Resend	Transactional mail out

Two operational consequences of the seeding scheme: the one-shot config_seed init container copies the repo app.ini into the data volume only if none exists, and runner_seed regenerates the runner config from its template on every deploy - so editing app.ini in the repo does not change a running instance; the seeded file in the volume is the live one, and GITEA__ / FORGEJO__ env vars override it at start.

Backups and the restore drill

A router systemd timer fires at 02:30 with up to 10 minutes of random delay. The unit runs pg_dump -Fc, checks the dump for the PGDMP magic, renames it atomically, rsyncs the bare-repo tree, then rsyncs both to the storage host’s backup dataset over ssh in push mode.

Copy history comes from sanoid snapshots of the backup dataset, not from dated files. Per-run git bundles were rejected on size: ~6.3GB times 7-day retention is ~44GB per box.

Last run on 2026-09-29: 02:38:07 to 02:38:23 +08, Result=success, 15.7s wall clock, 19.7M sent.

The restore drill passed on 2026-09-23, recorded in the migration plan rather than re-run for this page: a clone of a mirrored bare repo on the storage host matched the live HEAD. The Postgres dump was mounted into a scratch Postgres container (piping it over ssh ... docker run -i lost the stream and pg_restore saw no magic), and the full restore ran with 0 errors, 130 TABLE DATA entries, and 1 user + 268 repositories, matching live at the time. The timer and snapshot pattern is the same one the rest of the storage host uses, covered in Declarative backups on a ZFS homelab: nix timers, sanoid, syncoid.

Push mirrors to GitHub

206 push mirrors exist, one per mirrored repo. A push mirror syncs periodically and, with “Sync when new commits are pushed”, as soon as there are changes; it mirrors branches, tags, and commits; LFS objects are not mirrored; a GitHub push mirror needs a token with repository Contents permission, plus Workflows if .github/workflows exists.² Every mirror in the fleet has the 8h interval and sync-on-commit set, verified by a group by over the push_mirror table on 2026-09-29.

Forgejo 16 rejects credentials embedded in the mirror URL. The GitHub token is stored in each push mirror’s username/password fields, and Forgejo never returns the password, so which token the fleet uses cannot be read back from Forgejo.

GitHub runs no Actions for the mirrored repos; the Actions unit is disabled per repo, so the backup stays a git-only copy.

Which repos get a mirror is a policy, settled on 2026-09-30. Every active repo of the owner’s gets a push mirror. An archived repo keeps its GitHub copy only if one already existed, and three archived repos stay Forgejo-only by choice. Unmaintained third-party imports (36 of them that day) are archived on Forgejo with Actions off and no mirror. The OpenWrt fork is deliberately unmirrored, because it tracks upstream. The same-day tidy enrolled two active repos that had been missed, with HEADs verified equal on both sides, which brought the count to 208.

Two behaviours of the mirror are easy to trip over:

A branch deleted on Forgejo is not deleted on GitHub. The push mirror pushes the refs that exist and leaves the others in place, so a branch cleanup has to be repeated on GitHub by hand. On 2026-09-29 that meant six branches across four repos.
A push mirror on an archived repo keeps retrying and fails once the GitHub side moves on (one archived repo carried a PushRejected last_error this way). Removing that mirror is itself a write, so an archived repo answers the delete with 423 until it is unarchived; the order is unarchive, delete, re-archive.

Deploy path

Composer (GitOps) redeploys a stack when a webhook fires; for this stack it runs docker compose -f docker-compose.router.yml up -d.

The forgejo stack itself is deliberately sourced from GitHub, not from Forgejo: if Composer pulled the forge’s config from the forge, a forge outage could not be redeployed (bootstrap circle). The flow is push to Forgejo -> push mirror lands on GitHub -> the GitHub webhook fires Composer -> Composer syncs from GitHub. Verified on 2026-09-29: the stack’s repo URL on Composer is github.com and its webhook provider is github only.

All 14 other webhook-carrying stacks are sourced from Forgejo over SSH on 2223 (the last one was moved off GitHub on 2026-09-29). That is not circular, because repo git data is served from the router volume, not from Silo. The router’s NixOS config and deploy path live in Three NixOS hosts, one deploy interface.

Decision guide

In list form:

CI on your own hardware, and a git copy on GitHub is an acceptable outcome in a box outage: Forgejo primary + GitHub push mirror.
Development must continue on GitHub during a box outage, or GitHub Actions covers CI: GitHub primary + Forgejo pull mirror.
If a deploy tool pulls from the forge, keep the forge’s own stack sourced from GitHub; a forge outage must not block redeploying the forge.

Three NixOS hosts, one deploy interface - the edge router’s NixOS config, make deploy, and per-host deploy keys; host clones now fetch from Forgejo.
Declarative backups on a ZFS homelab: nix timers, sanoid, syncoid - the storage host’s pg_dump timers + sanoid snapshots pattern that the forge backup lands in.
knotea: one binary that is both your recursive resolver and your authoritative DNS - the Knot authority that answers the RFC 2136 DNS-01 challenge.
Hot vs bulk: placing app state on a two-tier ZFS homelab - where Silo’s data and the backup dataset sit on the storage host.

The runner and deploy wiring each have their own page: the Forgejo Actions runner (privileged posture, capacity, cache, liveness restart) and Forgejo webhooks to Composer GitOps (provider, allowed hosts, bootstrap circle, moving a stack’s source). Releases and the registry decision, storage, backups and caches, recovery runbooks and the fleet migration have their own pages too.

References

Forgejo, “Configuration Cheat Sheet,” Forgejo Documentation. https://forgejo.org/docs/latest/admin/config-cheat-sheet/ ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷
Forgejo, “Repository Mirrors,” Forgejo Documentation. https://forgejo.org/docs/latest/user/repo-mirror/ ↩ ↩²
Forgejo, “Storage settings,” Forgejo Documentation. https://forgejo.org/docs/latest/admin/setup/storage/ ↩ ↩²
Forgejo, “Utilizing Docker within Actions,” Forgejo Documentation. https://forgejo.org/docs/latest/admin/actions/docker-access/ ↩
Forgejo, “Securing Forgejo Actions Deployments,” Forgejo Documentation. https://forgejo.org/docs/latest/admin/actions/security/ ↩
Forgejo, “Forgejo Actions administrator guide,” Forgejo Documentation. https://forgejo.org/docs/latest/admin/actions/ ↩