Networking and DNSreference
Edge Caddy as a native NixOS service
The edge reverse proxy on the MS-01 edge router runs Caddy as a native NixOS systemd service, terminating public TLS, enforcing L4/L7 traffic tiers, and routing requests to internal homelab workloads. The binary is built with custom Go plugins via a custom Nix derivation that replaces Go module dependencies from local flake inputs rather than relying on public proxies. This reference documents the plugin packaging mechanics, the dependency override discipline, the September 2026 WAF simplification, the edgectl management control plane, and durable traffic routing patterns.
The router deployment transitioned from Docker Compose to native NixOS services on 2026-09-19. The containerised stack remains maintained in deploy/edge/ as an inactive deployment alternative, sharing the same Go plugin architecture and configuration concepts.
- Caddy runs as a native systemd service (
caddy.service) on the NixOS router, listening directly on host ports 80, 443, 2222, and 2223 withCAP_NET_BIND_SERVICEandCAP_NET_ADMIN. - The binary is packaged by
pkgs/caddy-edge.nixwithoutpkgs.caddy.withPluginsbecause the standard helper cannot resolve private local plugins. Instead, a custom derivation uses Go module directory replacement against flake inputs.1 - Public plugins (
rfc2136,cloudflare,caddy-l4,caddy-dynamicdns,cache-handler,storages/nuts/caddy) are pinned to exact versions; private plugins (caddy-policy-engine,caddy-body-matcher,caddy-ddos-mitigator) are replaced from local checkouts.2 - Transitive dependencies are strictly overridden via
-replacedirectives ingo.mod:grpcis pinned tov1.84.0andgolang.org/x/cryptotov0.57.0for security advisories, whilecel-gois deliberately held atv0.28.1because Caddy 2.11.4 is incompatible with CEL-Go 0.29+.345 - On 2026-09-26, the OWASP Core Rule Set (CRS), Coraza SecRules, and challenge interstitial layers were removed after traffic auditing confirmed zero blocking value and severe operational overhead. The in-house policy engine was preserved for targeted, hand-written rules.
- The
edgectlcontrol plane runs alongside Caddy as a native service on port 8082, managing dynamic IPsum blocklists, GeoIP lookups, and unified access log parsing backed by PostgreSQL 18 and Valkey.
Architecture
Section titled “Architecture”The request and control flow operates across five discrete steps:
- Inbound traffic on ports 80, 443, 2222, and 2223 terminates on Caddy running with ambient capabilities on the host network.
- The custom
caddy-ddos-mitigatorandcaddy-policy-engineHTTP handlers evaluate requests against active jail and rule files hot-reloaded from/var/lib/caddy/waf/. - Allowed requests pass through to internal backends across local subnets, macvlan bridges, or L4 TCP streams.
- Caddy emits structured JSON logs with security metadata injected via
log_appendto/var/log/caddy/combined-access.log. - The
edgectlsidecar tails access logs, updates security event analytics in PostgreSQL, downloads daily IPsum feeds, and rewrites the rules and jail files read by Caddy.
Plugin build architecture
Section titled “Plugin build architecture”In standard NixOS environments, custom Caddy binaries are built with pkgs.caddy.withPlugins. That helper accepts module names and versions, invoking xcaddy to download and compile modules from the public Go proxy (proxy.golang.org).
This mechanism fails when packaging custom in-house plugins that live in private repositories or local flake inputs. pkgs.caddy.withPlugins cannot access private authentication tokens during the build phase and cannot accept arbitrary filesystem paths for module code.
To resolve this, pkgs/caddy-edge/default.nix replaces the standard helper with a sandboxed buildGoModule derivation. The module graph is resolved once, off the build, by pkgs/caddy-edge/update.sh, which commits go.mod and go.sum next to the derivation:
(buildGoModule.override { go = go_1_26_10; }) { pname = "caddy-edge"; version = "2.11.4"; src = fetchFromGitHub { owner = "caddyserver"; repo = "caddy"; tag = "v2.11.4"; hash = "..."; };
# Runs in both the vendor fixed-output derivation and the build. postPatch = '' cp ${./go.mod} go.mod cp ${./go.sum} go.sum mkdir -p plugged private cp ${./main.go} plugged/main.go cp -r ${bodyMatcher} private/bodyMatcher cp -r ${policyEngine} private/policyEngine cp -r ${ddosMitigator} private/ddosMitigator cp -r ${souinFork} private/souinFork chmod -R u+w private '';
vendorHash = "sha256-..."; subPackages = [ "plugged" ];}Build mechanics and entrypoint isolation
Section titled “Build mechanics and entrypoint isolation”The derivation addresses several Go module packaging constraints:
- Subdirectory entrypoint: The build creates a
plugged/main.gofile inside a subdirectory rather than at the root of the source tree. Because the root ofcaddyserver/caddydeclarespackage caddyas a library, placing apackage mainentrypoint at the root breaks Go module parsing. The entrypoint importsgithub.com/caddyserver/caddy/v2/cmdto run the standard CLI interface. - Explicit standard modules: The entrypoint must import
_ "github.com/caddyserver/caddy/v2/modules/standard".2 Without this explicit blank import, Caddy registers only approximately 45 base modules out of roughly 197 standard modules. Basic directives such asfile_server,reverse_proxy,encode, and standard log encoders fail during Caddyfile adaptation. - Relative directory replacement: Private plugins are flake inputs, copied into the source tree at
./private/<input>. The committedgo.modreplaces each private module with that relative path:Terminal window go mod edit \-replace github.com/erfianugrah/caddy-policy-engine=./private/policyEngine \-replace github.com/erfianugrah/caddy-body-matcher=./private/bodyMatcher \-replace github.com/erfianugrah/caddy-ddos-mitigator=./private/ddosMitigator \-replace github.com/darkweak/souin=./private/souinFork
A replace pointing at a /nix/store path would end up in vendor/modules.txt, and a fixed-output derivation may not reference store paths. The consequence is that vendorHash covers the private plugin sources, so every plugin bump needs a new hash.
4. No network in the build: Until 2026-11-05 the derivation ran go get and go mod tidy during buildPhase, which needed __noChroot = true; plus nix.settings.sandbox = "relaxed" on the router. The output was then not a function of its inputs, and an outage of the Go module proxy or GitHub failed the router deploy. The network step now lives in update.sh; the build only reads the vendored module set. The same change gave the build its own Go 1.26.10 (a go.overrideAttrs on the pinned 1.26.5), because the old GOTOOLCHAIN=go1.26.10 download is impossible in the sandbox.
Module inventory
Section titled “Module inventory”The resulting binary packages six public plugins and three private plugins:
| Plugin | Source / Pin | Purpose |
|---|---|---|
caddy-policy-engine | Private flake input | L7 rule evaluation, header filtering, honeypots |
caddy-body-matcher | Private flake input | HTTP request body matching and variable extraction |
caddy-ddos-mitigator | Private flake input | Adaptive behavioral rate limiting and kernel jail management |
souin | Private fork | RFC-compliant HTTP cache handler |
caddy-dns/rfc2136 | v1.0.0 | RFC 2136 DNS-01 dynamic updates to local Knot DNS6 |
caddy-dns/cloudflare | v0.2.4 | Cloudflare DNS-01 fallback for public hostnames |
mholt/caddy-l4 | v0.1.2 | Layer 4 TCP proxying (SSH passthrough) |
mholt/caddy-dynamicdns | a5890c9 | Dynamic DNS record synchronization |
caddyserver/cache-handler | v0.16.0 | Standard HTTP cache directive integration |
storages/nuts/caddy | v0.0.19 | NutsDB storage backend for cache handlers |
Dependency overrides and the CEL-Go hold
Section titled “Dependency overrides and the CEL-Go hold”Transitive Go dependencies pulled in by plugins can introduce build breakage or security vulnerabilities. The Nix packaging enforces explicit version overrides to maintain reproducible builds and security compliance.
The 2026-09-29 security replacements
Section titled “The 2026-09-29 security replacements”On 2026-09-29, fleet vulnerability audits identified open Dependabot alerts across transitive dependencies in caddy-policy-engine and caddy-ddos-mitigator. update.sh writes explicit -replace directives into the committed go.mod:
go mod edit \ -replace google.golang.org/grpc=google.golang.org/grpc@v1.84.0 \ -replace golang.org/x/crypto=golang.org/x/crypto@v0.57.0 \ -replace golang.org/x/net=golang.org/x/net@v0.60.0- gRPC
v1.84.0: Replacedv1.82.1to clear high-severity gRPC transport vulnerabilities resolved in upstream releases 1.82.2 and 1.83.1.5 - x/crypto
v0.57.0: Bumped to satisfy security scanner requirements regarding SSH channel denial-of-service vulnerabilities. - x/net
v0.60.0: Added 2026-10-22 for CVE-2026-78669. - No x/text replace: an earlier
v0.39.0pin was removed on 2026-10-22. The module graph resolvesv0.42.0on its own, so the pin had become a downgrade into CVE-2026-56851.
The CEL-Go v0.28.1 hold
Section titled “The CEL-Go v0.28.1 hold”The Common Expression Language (cel-go) dependency in caddy-policy-engine is deliberately held at version v0.28.1. Upgrading cel-go to v0.29.0 or later breaks compilation of Caddy 2.11.4:
cannot use fn (value of type func(interpreter.InterpretableV2) interpreter.InterpretableV2)as interpreter.InterpretableDecorator in argument to interpreter.NewCallIn cel-go 0.29, the internal interpreter.NewCall signature was refactored to require InterpretableV2, a breaking API change that Caddy core has not yet adopted.3
Security scanners report advisory GO-2026-6094 against cel-go v0.28.1.4 Analysis of the vulnerability confirms it is non-actionable for this proxy: GO-2026-6094 describes a panic condition when evaluating expressions involving NativeTypes. Caddy’s HTTP expression matchers evaluate standard string, path, and header attributes, and never enable or execute NativeTypes evaluation. Holding at v0.28.1 preserves stable builds without exposing the runtime to vulnerability triggers.
What was removed in the September 2026 redesign
Section titled “What was removed in the September 2026 redesign”Between 2026-09-23 and 2026-09-26, the edge proxy underwent an audit and architectural simplification. The preceding stack incorporated the OWASP Core Rule Set (CRS v4) via Coraza rules, an automated SecRule translation pipeline, and proof-of-work browser challenges.
Preflight audit findings
Section titled “Preflight audit findings”A 30-day preflight traffic audit (docs/nuke-snapshot-2026-09-23/) across seven production sites revealed a severe disproportion between complexity and defensive value:
- 84,373 total policy events were logged across 1,013 unique client IP addresses.
- 0 actual requests were blocked. The entire CRS rule corpus was operating in detection-only mode because enforcing it caused false positives on legitimate client APIs.
- 342 imported CRS rules generated thousands of notice-level events (such as rule
9100034request body inspection and rule930130scanner user-agent matches), drowning genuine operational signals in noise. - One custom rule: A single exclusion rule skipping false positives for the Atuin shell history sync client accounted for the only critical policy configuration.
- Maintenance overhead: Maintaining the
tools/crs-convertertranslation pipeline and its 4,566-test regression suite required ongoing debugging, with 93 standing CRS conversion regressions remaining unresolved.
Architectural pruning
Section titled “Architectural pruning”On 2026-09-26, the stack was simplified to eliminate unneeded components:
| Component | Status | Rationale |
|---|---|---|
| OWASP Core Rule Set (CRS) | Removed | 342 generic rules produced noise; 0 blocks across 84k events |
| Coraza SecRule engine | Retired | Replaced by native Go caddy-policy-engine handler |
coraza-caddy fork | Archived | Fork retired 2026-09-29; archived on Forgejo |
tools/crs-converter | Removed | Eliminated 4,566-test converter test suite and standing regressions |
| Challenge interstitial | Removed | SHA-256 hashcash PoW and service worker tracking removed from edge path |
| WAF UI settings pages | Pruned | Removed 9 complex configuration views from the dashboard |
An initial cleanup attempt on 2026-09-26 removed the entire WAF subsystem. This was immediately corrected: the removal target was CRS and Coraza rule translation, not the underlying policy engine. The in-house caddy-policy-engine was restored to evaluate lightweight, hand-written rules (such as honeypots, path deny-lists, and Atuin exceptions) without the operational burden of third-party signature sets.
The edgectl control plane
Section titled “The edgectl control plane”edgectl (renamed from wafctl on 2026-09-27) is the companion management daemon running alongside Caddy on the router. It operates as a native systemd service (edgectl.service) listening on 127.0.0.1:8082 (leaving host port 8080 dedicated to Composer).
The control plane interaction handles data persistence and service boundaries:
- Permission isolation:
edgectlruns under its own system user (edgectl) but belongs to supplementary groupcaddy. The directory/var/lib/caddy/wafis set to mode2775(setgidcaddy), allowingedgectlto write configuration files thatcaddyreads. - Storage transition: On 2026-09-30,
edgectlcut over from flat JSON files to PostgreSQL 18 (postgres:///edgectl?host=/run/postgresql) via Unix socket peer authentication. PostgreSQL stores rule definitions, managed lists, and security events. A dedicated local Valkey instance (/run/edgectl-valkey/valkey.sock) mirrors active jail entries. - Decoupled file sync: Caddy does not communicate with PostgreSQL or Valkey directly.
edgectlexports compiled rule sets to/var/lib/caddy/waf/policy-rules.jsonand active IP jail entries to/var/lib/caddy/waf/jail.json. Caddy hot-reloads these files asynchronously. Ifedgectlor the database stops, Caddy continues serving traffic using the last written state on disk. - Dashboard interface: The static frontend (Astro 7 and React 19) is built by Nix from the same pinned
caddy-composerevision asedgectl(pkgs/edgectl-ui.nix,buildNpmPackage), andEDGE_UI_DIRpoints at its store path. Caddy reverse proxies the LAN-only dashboard hostname toedgectl, which serves both the API and the static files. Before 2026-11-05 the files were copied by hand out of the CI image into/var/lib/edgectl/ui, so a rebuilt router came up without a dashboard and the live copy drifted: it still carried nine pages the September pruning had removed from the source.
Durable runtime mechanics
Section titled “Durable runtime mechanics”The native NixOS deployment carries forward core traffic management, threat intelligence, and logging patterns established in the edge architecture.
Traffic tiers in Caddyfile
Section titled “Traffic tiers in Caddyfile”Site definitions in the router’s edge/Caddyfile compose modular snippets to enforce traffic tiers:
(ddos) { ddos_mitigator { jail_file /var/lib/caddy/waf/jail.json threshold 0.65 base_penalty 60s max_penalty 24h warmup_requests 1000 }}
(waf) { import ddos policy_engine { rules_file /var/lib/caddy/waf/policy-rules.json reload_interval 5s } handle_errors 400 403 429 { import error_pages }}
(lan_only) { @denied not remote_ip 10.0.0.0/8 100.64.0.0/10 172.16.0.0/12 192.168.0.0/16 respond @denied 401}
(research_auth) { @denied { not remote_ip 10.0.0.0/8 100.64.0.0/10 172.16.0.0/12 192.168.0.0/16 not header Authorization "Bearer {$RESEARCH_TOKEN}" } respond @denied 401}The snippets partition services into four access tiers:
- Public services with WAF (
import waf): Evaluatescaddy-ddos-mitigatoranomaly scoring andcaddy-policy-enginerules on internet-facing endpoints such as the Forgejo host and the resolver’s DoH endpoint. - LAN and Tailnet restricted (
import lan_only): Enforces CIDR filtering allowing only RFC 1918 private subnets and Tailscale CGNAT addresses (100.64.0.0/10). Applied to internal administrative consoles such as the edgectl dashboard and the traffic-analysis UI. - Token authenticated (
import research_auth): Bypasses authentication for LAN and tailnet clients, but requires a pre-shared bearer token ({$RESEARCH_TOKEN}) for external requests. Applied to compute services such as the search and crawler endpoints of the research stack. - Layer 4 SSH passthrough: Caddy uses
caddy-l4to proxy raw TCP traffic for port 2222 (forwarded to the documentation SSH daemon on port 2222) and port 2223 (forwarded to Forgejo’s container SSH server on port 2222).
IPsum blocklist lifecycle
Section titled “IPsum blocklist lifecycle”Malicious IP defense relies on the stamparm/ipsum threat intelligence feed, aggregate data compiled from over 30 public threat lists:7
- Daily retrieval:
edgectldownloadsipsum.txtdaily at 06:00 UTC (configured byEDGE_BLOCKLIST_REFRESH_HOUR=6). Operators can also trigger a refresh on demand from the dashboard. - Score grouping: IPs are parsed and categorised across all eight threat score levels (1 through 8). IPs with a score of 1 or higher are ingested into eight managed lists (
ipsum_level_1throughipsum_level_8). - Rule synchronization:
edgectlcompiles these lists into policy engine block rules namedIPsum Block (Level N). - Execution: The rules are written to
policy-rules.json. When a matching IP connects, Caddy’s policy engine blocks the request with a 403 Forbidden before backend evaluation occurs.
Three-tier GeoIP resolution
Section titled “Three-tier GeoIP resolution”For security analytics and geographic rule matching, edgectl resolves client IPs through a priority chain:
Priority 1: Cf-Ipcountry header (0 ms latency, injected by Cloudflare CDN)Priority 2: Local MMDB lookup (<1 µs, pure-Go reader against /var/lib/edgectl/geoip/country.mmdb)Priority 3: Online API fallback (cached in memory with an LRU cache, 24h TTL)The local MMDB lookup uses a pure-Go MaxMind reader requiring no CGO dependencies or external shared libraries. If the Cloudflare header is absent and the local database does not match, the query falls back to a rate-limited online lookup.
Unified access log parsing
Section titled “Unified access log parsing”Instead of maintaining separate web server and security audit logs, Caddy writes a single unified access stream to /var/log/caddy/combined-access.log. Security metadata is injected using log_append directives in the (site_log) snippet:
(site_log) { log_append request_id {http.request.uuid} log_append ddos_action {http.vars.ddos_mitigator.action} log_append ddos_fingerprint {http.vars.ddos_mitigator.fingerprint} log_append ddos_z_score {http.vars.ddos_mitigator.z_score} log_append ddos_spike_mode {http.vars.ddos_mitigator.spike_mode} log_append policy_action {http.vars.policy_engine.action} log_append policy_rule {http.vars.policy_engine.rule_name} log_append policy_ja4 {http.vars.policy_engine.ja4} log_append policy_score {http.vars.policy_engine.anomaly_score} log_append policy_detect_rules {http.vars.policy_engine.detect_rules} log_append policy_detect_matches {http.vars.policy_engine.detect_matches} log combined { level INFO output file /var/log/caddy/combined-access.log { roll_size 100MiB roll_keep 10 } format json { time_format wall } }}edgectl’s AccessLogStore tails /var/log/caddy/combined-access.log using byte offset tracking saved to /var/lib/edgectl/.access-log-offset. When log rotation occurs, the file size drops below the saved offset; edgectl detects the truncation, resets the offset to zero, and resumes streaming without missing events or re-parsing rolled logs.
DDoS mitigator plugin internals
Section titled “DDoS mitigator plugin internals”The DDoS mitigator (http.handlers.ddos_mitigator) runs before the policy engine in Caddy’s handler chain. The Caddyfile snippet in the traffic tiers section places ddos_mitigator outside policy_engine so blocked traffic never reaches downstream WAF processing, saving CPU under load. The plugin header comment states the ordering explicitly: “log_append runs first (outermost), then ddos_mitigator, then policy_engine.”8
Three-layer detection architecture
Section titled “Three-layer detection architecture”The plugin scores every request through three independent layers, then jails IPs whose score crosses a configurable threshold (default 0.75):
Layer 1 - Global rate gate: Per-IP sustained requests per second across all services, measured over a 60-second sliding window using a 64-slot ring buffer of request timestamps.9 When global_rate_threshold is set (default 0, disabled) and the rate exceeds it, the IP is jailed immediately regardless of path diversity. This catches server-saturating floods that spread across many paths.
Layer 2 - Per-service behavioral profiling: An ipTracker keyed on (IP, host) gives each service its own behavioral profile. Path diversity - unique paths divided by total requests - detects targeted low-rate attacks: a normal user browsing 16 pages scores 1.00, while a flood hitting one path repeatedly scores below 0.01. Volume confidence dampens low-request-count profiles, and rate amplification raises the score when a 60-second window rate is high with low diversity.10
Layer 3 - Host diversity exculpation: A global per-IP host count from hostTracker reduces the effective score at jail-decision time: effectiveScore = rawScore / log2(uniqueHosts + 1).11 A real user hitting 8 services gets a roughly 3.17x score reduction; a DDoS targeting one service gets none. The min_host_exculpation setting (default 2) sets a floor below which dampening does not activate, so first-time visitors to a single host are not prematurely excused.
Request hot path
Section titled “Request hot path”The ServeHTTP method runs six guarded steps on every request before reaching the backend:
- Whitelist check - private subnet addresses pass immediately.
- Jail and promoted-CIDR check - 64-shard concurrent map, O(1) per request via
IsJailed. If jailed and the adaptive stats engine is in spike mode, only 1 in 64 blocked requests is logged to avoid the synchronous JSON log pipeline becoming the bottleneck.12 - Host recording and L1 rate check - global per-IP ring buffer rate compared to threshold.
- L2 and L3 behavioral scoring -
RecordAndScoreon the(IP, host)profile, then divide by host diversity factor. - Jail decision - if score exceeds threshold, the IP is added to the jail with exponential backoff (
base_penalty * 2^infractions) and up to 25% jitter to prevent synchronised retry storms.13 The CIDR aggregator checks whether enough IPs from the same /24 (IPv4) or /64 (IPv6) have been jailed to promote the entire prefix. - Pass through -
next.ServeHTTPfor clean requests.
The hot path benchmarks at approximately 90 ns per request with zero heap allocation.14
State and lifecycle
Section titled “State and lifecycle”The plugin is provisioned once per Caddy process and holds all state in memory:
- ipJail: 64-shard concurrent map (
sync.RWMutexper shard) with TTL-based lazy expiry. A package-level registry maps jail file paths to shared jail instances, so the L4 TCP RST handler (mitigator_l4.go) accesses the same jail as the L7 handler without any IPC.15 - Infraction history: 24-hour TTL keyed by IP. Counts persist across jail terms so repeat offenders escalate penalties even after expiry or manual unjail.16
- hostTracker: Per-IP host diversity tracked globally across all services. The host count and ring buffer rate are updated atomically on each request.
- ipTracker: Per-
(IP, host)behavioral profiles with LRU eviction. Each profile stores a ring buffer for the 60-second rate, a bounded set of up to 256 unique paths, and HTTP method counts. - Count-Min Sketch: Fingerprint frequency tracking across the whole traffic corpus. Counters are halved every
decay_interval(default 30 s) so the sketch responds to traffic shifts. - adaptiveStats: Welford algorithm plus dual exponential weighted moving averages for spike detection. Below
warmup_requests(default 1000), the stats engine suppresses scoring to prevent false positives during cold start.17 - CIDR aggregator: Tracks jailed IPs per /24 and /64 prefix. When
cidr_threshold_v4(default 5) IPs are jailed from one prefix, the entire prefix is promoted and any subsequent connection from that range is blocked instantly without per-IP lookup.
Enforcement layers
Section titled “Enforcement layers”Four enforcement mechanisms share the same jail, deployed from fastest (kernel) to slowest (userspace):
| Layer | Mechanism | Config flag | Capability required |
|---|---|---|---|
| eBPF/XDP | XDP_DROP at NIC driver, approximately 10M packets per second | xdp_drop, xdp_iface | BPF, NET_ADMIN |
| nftables | Kernel ipset drop, approximately 1.5M packets per second | kernel_drop | NET_ADMIN |
| L4 TCP | SetLinger(0) sends RST before TLS handshake | Via caddy-l4 app | NET_BIND_SERVICE |
| L7 HTTP | 403 Forbidden, approximately 50K requests per second | Default, always active | None |
nftables uses the google/nftables library with auto-reconnection on netlink failure and tear-down of all rules, chains and tables on Caddy shutdown.18 XDP loads a BPF program via cilium/ebpf; the sync uses diff-based updates (only changed entries) rather than full reloads. Both kernel layers log a warning and fall back gracefully if the required capability is missing at Provision time.19
Config reload behaviour
Section titled “Config reload behaviour”Configuration is set at Provision time via Caddyfile or JSON and does not hot-reload. The jail file is the dynamic surface: the runFileSync goroutine reads and writes jail.json every sync_interval (default 2 s) under flock.20 On read, new entries from edgectl are merged in without removing existing ones. When an entry present in memory is absent from the file and predates the file’s updated_at timestamp, the plugin treats it as a manual unjail: the IP is removed from the jail, the CIDR counter is decremented, and a 60-second grace period prevents immediate re-jail from stale profile data accumulated before the unjail.21
Infraction history is written to the jail on jail.Add and persisted in memory separately from the jail map. It survives expiry and manual unjail so a repeat offender escalates from its last known infraction count rather than starting over.
Policy engine plugin internals
Section titled “Policy engine plugin internals”The policy engine (http.handlers.policy_engine) evaluates user-defined rules through a 7-pass priority pipeline. The order of passes is determined by numeric priority bands: rules within each band are sorted by priority then by rule ID for deterministic tie-breaking.22
The engine implements six Caddy interfaces: Module, Provisioner, Validator, CleanerUpper, MiddlewareHandler (ServeHTTP), and Unmarshaler for Caddyfile parsing.23 A JA4 TLS fingerprinting listener wrapper (caddy.listeners.ja4) computes fingerprints at connection-accept time for use in rule conditions.
7-pass evaluation pipeline
Section titled “7-pass evaluation pipeline”| Pass | Priority band | Rule type | Behaviour |
|---|---|---|---|
| 1 | 50-99 | allow | Short-circuit: skip all remaining passes, pass to next handler immediately. |
| 2 | 100-149 | block, honeypot | Return 403 Forbidden. In detection_only mode, log but do not reject. |
| 3 | 150-199 | challenge | Verify PoW cookie or serve interstitial HTML. |
| 4 | 200-299 | skip | Accumulate skip flags for downstream passes (detect, rate_limit, block, challenge). Non-terminating. |
| 5 | 300-399 | rate_limit | Sliding window counter per rule-defined key. Returns 429 on exceed, or logs only in log_only action. |
| 6 | 400-499 | detect | Anomaly scoring (inbound and outbound). If accumulated score exceeds configured threshold at end of loop, return 403. |
| 7 | 500-599 | response_header | Set, add, remove, or default-set response headers after upstream processing. |
The inbound loop (passes 1 through 6) runs before next.ServeHTTP. Pass 7 and outbound detect rules run after the upstream response. The ServeHTTP implementation wraps the response writer to capture status code and body for outbound matching.24
An allow rule that matches still lets rate_limit rules evaluate for allowed traffic, so rate limits are enforced regardless of allow status.25 A skip rule can suppress specific rule IDs, entire passes, or all remaining evaluation altogether via skip_targets.all_remaining.
Rule compilation and hot-reload
Section titled “Rule compilation and hot-reload”Rules are loaded from policy-rules.json as a JSON file generated by edgectl. At load time, every rule is compiled into a compiledRule struct: regular expressions are pre-compiled, operator names are resolved to enums, byte-range validation sets are pre-built as 256-bool arrays, challenge difficulty is bounded, and an needsBody flag is set for rules that reference request body fields.26
Hot-reload runs via mtime polling at the configured reload_interval (default 5 s). The checkReload function stats the user rules file, the optional default rules file, and the JTI denylist file. On any mtime change, it opens the file under a single os.Open descriptor, stats the same descriptor to avoid TOCTOU races, and reads and parses the JSON content.27 If compilation fails, the old rules remain in place. If the file is deleted, rules are cleared and rate limit zones are emptied.28
Rate limit counter state is preserved across reloads for zones whose configuration (events and window) has not changed. Zones whose parameters change get fresh counters; removed zones are evicted.29
State and lifecycle
Section titled “State and lifecycle”- Compiled rules: Stored behind a
sync.RWMutex(read-locked duringServeHTTP, write-locked during reload). Rule count, enabled count, and rate-limit zone count are logged at Provision. - Rate limit state: 16-shard concurrent maps using fixed-window interpolation (
prevCount * (1 - elapsed/window) + currCount), matching the algorithm used by nginx and envoy.30 A background sweep goroutine evicts expired counters every 30 s. Max keys per zone defaults to 100000 to bound memory growth. - JA4 registry: A package-level
sync.Mapkeyed by remote address. The listener wrapper writes fingerprints atAccept()time;ServeHTTPreads them. The wrapper uses a 2-second peek deadline because upstream wrappers such ascaddy-l4clear the connection deadline after route matching - without it, a silent or drip-feeding client parked the serial accept loop during the 2026-08-08 edge outage.31 - Challenge state: A 32-byte HMAC-SHA256 key for cookie signing, a flag indicating whether any challenge rules exist in the loaded config, and an mtime-polled JTI denylist file for retrospective cookie invalidation.32
- WAF config: Per-service anomaly scoring thresholds, protocol enforcement limits, and disabled detect-category lists, resolved by host at request time.
Challenge system
Section titled “Challenge system”The challenge system serves an embedded HTML interstitial page when a challenge rule matches and no valid cookie is present. Client-side JavaScript spawns a Web Worker that performs SHA-256 hashcash proof-of-work, collects 13 browser-probe signals (webdriver, CDC markers, WebGL renderer, audio context, speech voices, permissions timing, languages, chrome.runtime, platform, memory, touch, screen), and sends the solution to /.well-known/policy-challenge/verify.33 The server validates the HMAC, PoW hash, and timing, then runs a 6-layer bot scoring pass. A score below 70 issues an HMAC-signed cookie with configurable TTL that lets the client bypass subsequent challenge rules.
Adaptive difficulty maps a pre-solve signal score linearly to a difficulty range configured per rule (min_difficulty to max_difficulty).34 A session tracking service worker and page-level collector inject beacons for cross-navigation correlation. The JTI denylist enables invalidation of specific cookies without rotating the global HMAC key.
CRS protocol enforcement
Section titled “CRS protocol enforcement”Before the main rule loop, enforceProtocolLimits() checks request-level properties against configurable limits: HTTP method allowlist, HTTP version allowlist, maximum argument count, argument name and value lengths, total query string length, and upload size limits.35 Violations produce synthetic detect matches with CRITICAL severity (+5 anomaly score) using their original CRS rule IDs for event correlation. All limits support per-service overrides via WafServiceConfig; a zero value means not enforced.
Config reload behaviour
Section titled “Config reload behaviour”Unlike the DDoS mitigator which syncs a jail file bidirectionally, the policy engine reads its rules file unidirectionally: edgectl writes, Caddy polls. A failed hot-reload logs the error and keeps the previous rule set. A removed rules file clears all rules and rate limit zones. The JTI denylist file is polled independently on the same ticker so cookie invalidation updates propagate without waiting for a rules change.
The rlState (rate limit state) pointer is set once at Provision and never set to nil, even during Cleanup, because ServeHTTP reads it without holding the rules mutex. Nil-ing it would race with in-flight requests.36
Operational gotchas
Section titled “Operational gotchas”If the JA4 listener wrapper’s Accept() blocks without its 2-second clientHelloPeekTimeout deadline, the serial accept loop stalls and no new TLS connections are accepted. The deadline is cleared after the peek so slow-but-legitimate TLS clients are not affected.
Challenge rules that bind cookies to the JA4 fingerprint (bind_ja4: true) fail when the client’s TLS fingerprint changes between the interstitial solve and the verification POST - for instance, a browser that rotates its cipher suite order. The cookie is rejected and the client is served a fresh interstitial.
Detection-only mode for block rules (detection_only in WAF config) turns 403s into logged events. The rate limiter has its own independent log_only action per zone. The two settings do not interact, so a block rule in detection-only mode still triggers a block_logged action variable while a rate limit rule with action: log_only produces a different variable path.
Decision guide: native service vs containerised stack
Section titled “Decision guide: native service vs containerised stack”Both deployment models are maintained in the repository workspace. Selecting between native NixOS and Docker Compose depends on host role and network architecture:
| Property | Native NixOS (caddy-edge.nix) | Containerised Compose (deploy/edge/) |
|---|---|---|
| Host role | Primary edge router (MS-01) | Standalone Docker host or secondary node |
| Network topology | Direct host binding (CAP_NET_BIND_SERVICE) | Docker bridge, host network, or macvlan |
| Binary delivery | Nix derivation with flake input replace | Multi-stage Dockerfile built via xcaddy |
| Service manager | Systemd units managed by NixOS | Docker Compose daemon |
| Storage layer | Local PostgreSQL 18 and Valkey unix sockets | Docker volume bind mounts or containerised DB |
| Operational state | Active production deployment | Inactive deployment alternative |
The decision flow follows three criteria:
- Deploy as a native NixOS service on the MS-01 edge router to avoid container bridge overhead, bind host ports cleanly with systemd capabilities, and manage dependencies through the flake lockfile.
- Deploy via Docker Compose on standard Linux distributions or container worker nodes where Nix package management is unavailable.
- Keep the Compose configuration updated in
deploy/edge/as a verified operational fallback if router services need isolation during host maintenance.
Related docs
Section titled “Related docs”- Self-hosted Forgejo on the router - How Forgejo runs on the same edge router, receiving SSH and HTTPS traffic forwarded through Caddy.
- Tailscale homelab network topology - Subnet routing and CGNAT IP address space referenced by Caddy’s
(lan_only)filter snippet. - Legacy containerised Caddy WAF guide - Historical reference for the 2026-02 Docker Compose and Coraza implementation.
References
Section titled “References”-
Caddyserver, “Extending Caddy,” Caddy Docs. https://caddyserver.com/docs/extending-caddy ↩
-
Caddyserver, “Standard Modules,” Caddy Docs. https://caddyserver.com/docs/modules/ ↩ ↩2
-
Common Expression Language, “CEL-Go Releases,” GitHub. https://github.com/google/cel-go/releases ↩ ↩2
-
Go Security Team, “GO-2026-6094: Panics when evaluating expressions involving NativeTypes in github.com/google/cel-go,” Go Vulnerability Database. https://pkg.go.dev/vuln/GO-2026-6094 ↩ ↩2
-
gRPC Project, “gRPC-Go Releases,” GitHub. https://github.com/grpc/grpc-go/releases ↩ ↩2
-
P. Vixie et al., “Dynamic Updates in the Domain Name System (DNS UPDATE),” RFC 2136, IETF. https://www.rfc-editor.org/rfc/rfc2136 ↩
-
Stamparm, “IPsum: Threat Intelligence Feed,” GitHub. https://github.com/stamparm/ipsum ↩
-
caddy-ddos-mitigator,
mitigator.golines 1-10 (package doc comment: handler ordering log_append, ddos_mitigator, then policy_engine). ↩ -
caddy-ddos-mitigator,
profile.golines 1-26 (package doc describing three-layer architecture) and lines 62-73 (rateWindowSize, rateWindowNs, ring buffer constants). ↩ -
caddy-ddos-mitigator,
profile.golines 139-147 (record method updating per-profile request counts, unique paths, and ring buffer on each request). ↩ -
caddy-ddos-mitigator,
mitigator.golines 478-488 (hostsForScore logic dividing by log2(uniqueHosts + 1) when host count meets MinHostExculpation). ↩ -
caddy-ddos-mitigator,
mitigator.golines 417-426 (spike-mode deterministic sampling: 1-in-64 blocked requests logged via FNV-32a hash). ↩ -
caddy-ddos-mitigator,
mitigator.golines 544-557 (calcTTLForInfractions with exponential backoff and up to 25% jitter). ↩ -
caddy-ddos-mitigator,
README.mdlines 271-277 (benchmark on AMD Ryzen 7 7800X3D, single-threaded: Jail IsJailed 9 ns/op, CMS Increment 8 ns/op, total hot path approximately 90 ns per request with zero allocation). ↩ -
caddy-ddos-mitigator,
jail.golines 270-284 (package-level jailRegistry map keyed by jail file path, getOrCreateJail function). ↩ -
caddy-ddos-mitigator,
jail.golines 190-200 (infraction history with 24-hour TTL, persists across jail terms). ↩ -
caddy-ddos-mitigator,
mitigator.golines 204-207 (WarmupRequests defaults to 1000, prevents false positives during startup and low-traffic periods). ↩ -
caddy-ddos-mitigator,
nftables.golines 1-26 (package doc: nftables ipset management, cleanup on shutdown, auto-reconnection on failure). ↩ -
caddy-ddos-mitigator,
mitigator.golines 245-255 (nftables setup: on NET_ADMIN unavailable, fall back to nftNoop). ↩ -
caddy-ddos-mitigator,
mitigator.golines 182-184 (SyncInterval default 2 s) and lines 667-760 (runFileSync with flock, bidirectional merge, and unjail detection). ↩ -
caddy-ddos-mitigator,
mitigator.golines 717-730 (60-second grace period on manual unjail). ↩ -
caddy-policy-engine,
AGENTS.mdlines 61-75 (7 rule types evaluated in priority-band order, sorted by priority then rule ID). ↩ -
caddy-policy-engine,
AGENTS.mdlines 267-280 (six Caddy interfaces: Module, Provisioner, Validator, CleanerUpper, MiddlewareHandler, Unmarshaler). ↩ -
caddy-policy-engine,
policyengine.golines 1385-1493 (outbound response wrapping: responseRecorder, ServeHTTP, then evaluateOutbound). ↩ -
caddy-policy-engine,
policyengine.golines 896-899 (allow-matched guard skips everything except rate_limit rules). ↩ -
caddy-policy-engine,
AGENTS.mdlines 159-161 (compiled counterparts pre-processed at load time for O(1) runtime lookups). ↩ -
caddy-policy-engine,
policyengine.golines 4771-4785 (loadFromFile: open and stat same file descriptor to avoid TOCTOU). ↩ -
caddy-policy-engine,
policyengine.golines 4708-4720 (checkReload: file removed clears rules and rate limit zones). ↩ -
caddy-policy-engine,
ratelimit.golines 1-14 (package doc: counter state preserved across hot-reloads, only zones whose config changes get fresh counters). ↩ -
caddy-policy-engine,
ratelimit.golines 7-11 (package doc: fixed-window interpolation matching nginx/envoy/cloudflare). ↩ -
caddy-policy-engine,
ja4_listener.golines 1-5 and 60-63 (package doc: 2-second peek deadline; unbounded peek parked serial accept loop during 2026-08-08 outage). ↩ -
caddy-policy-engine,
AGENTS.mdlines 220-223 (JTI denylist written by edgectl, polled by plugin for retrospective cookie invalidation). ↩ -
caddy-policy-engine,
AGENTS.mdlines 321-327 (6-layer bot scoring with 13 browser probes at L3). ↩ -
caddy-policy-engine,
challenge.golines 37-47 (ChallengeConfig fields: MinDifficulty, MaxDifficulty for adaptive range). ↩ -
caddy-policy-engine,
AGENTS.mdlines 82-103 (CRS protocol enforcement settings and their CRS rule IDs). ↩ -
caddy-policy-engine,
policyengine.golines 772-777 (Cleanup comment: rlState intentionally not nilled to avoid data race with in-flight ServeHTTP). ↩