Skip to content

Networking and DNSreference

Edge Caddy as a native NixOS service

The edge reverse proxy on the MS-01 edge router runs Caddy as a native NixOS systemd service, terminating public TLS, enforcing L4/L7 traffic tiers, and routing requests to internal homelab workloads. The binary is built with custom Go plugins via a custom Nix derivation that replaces Go module dependencies from local flake inputs rather than relying on public proxies. This reference documents the plugin packaging mechanics, the dependency override discipline, the September 2026 WAF simplification, the edgectl management control plane, and durable traffic routing patterns.

The router deployment transitioned from Docker Compose to native NixOS services on 2026-09-19. The containerised stack remains maintained in deploy/edge/ as an inactive deployment alternative, sharing the same Go plugin architecture and configuration concepts.

  • Caddy runs as a native systemd service (caddy.service) on the NixOS router, listening directly on host ports 80, 443, 2222, and 2223 with CAP_NET_BIND_SERVICE and CAP_NET_ADMIN.
  • The binary is packaged by pkgs/caddy-edge.nix without pkgs.caddy.withPlugins because the standard helper cannot resolve private local plugins. Instead, a custom derivation uses Go module directory replacement against flake inputs.1
  • Public plugins (rfc2136, cloudflare, caddy-l4, caddy-dynamicdns, cache-handler, storages/nuts/caddy) are pinned to exact versions; private plugins (caddy-policy-engine, caddy-body-matcher, caddy-ddos-mitigator) are replaced from local checkouts.2
  • Transitive dependencies are strictly overridden via -replace directives in go.mod: grpc is pinned to v1.84.0 and golang.org/x/crypto to v0.57.0 for security advisories, while cel-go is deliberately held at v0.28.1 because Caddy 2.11.4 is incompatible with CEL-Go 0.29+.345
  • On 2026-09-26, the OWASP Core Rule Set (CRS), Coraza SecRules, and challenge interstitial layers were removed after traffic auditing confirmed zero blocking value and severe operational overhead. The in-house policy engine was preserved for targeted, hand-written rules.
  • The edgectl control plane runs alongside Caddy as a native service on port 8082, managing dynamic IPsum blocklists, GeoIP lookups, and unified access log parsing backed by PostgreSQL 18 and Valkey.
MS-01 edge router (NixOS)Internet / WAN clientscaddy.service (caddy-edge):80, :443, :2222, :2223Plugins: policy-engine, ddos, body-matcherHTTPS / SSHLAN / Tailnet clientsHTTPS / SSH/var/log/caddy/combined-access.logstructured JSON + log_appendHomelab services(LAN macvlan, containers, storage NAS)reverse_proxy / l4edgectl.service (sidecar):8082 API + static dashboard/var/lib/edgectlPostgreSQL 18 (socket)postgres:///edgectlpeer authValkey (socket)IP jail mirrorunix socketincremental offset tailing/var/lib/caddy/waf/policy-rules.json(hot-reloaded every 5s)sync policy rules/var/lib/caddy/waf/jail.json(mtime polled)sync IPsum / jailevaluated in request pathevaluated by ddos-mitigator

The request and control flow operates across five discrete steps:

  1. Inbound traffic on ports 80, 443, 2222, and 2223 terminates on Caddy running with ambient capabilities on the host network.
  2. The custom caddy-ddos-mitigator and caddy-policy-engine HTTP handlers evaluate requests against active jail and rule files hot-reloaded from /var/lib/caddy/waf/.
  3. Allowed requests pass through to internal backends across local subnets, macvlan bridges, or L4 TCP streams.
  4. Caddy emits structured JSON logs with security metadata injected via log_append to /var/log/caddy/combined-access.log.
  5. The edgectl sidecar tails access logs, updates security event analytics in PostgreSQL, downloads daily IPsum feeds, and rewrites the rules and jail files read by Caddy.

In standard NixOS environments, custom Caddy binaries are built with pkgs.caddy.withPlugins. That helper accepts module names and versions, invoking xcaddy to download and compile modules from the public Go proxy (proxy.golang.org).

This mechanism fails when packaging custom in-house plugins that live in private repositories or local flake inputs. pkgs.caddy.withPlugins cannot access private authentication tokens during the build phase and cannot accept arbitrary filesystem paths for module code.

To resolve this, pkgs/caddy-edge/default.nix replaces the standard helper with a sandboxed buildGoModule derivation. The module graph is resolved once, off the build, by pkgs/caddy-edge/update.sh, which commits go.mod and go.sum next to the derivation:

(buildGoModule.override { go = go_1_26_10; }) {
pname = "caddy-edge";
version = "2.11.4";
src = fetchFromGitHub { owner = "caddyserver"; repo = "caddy"; tag = "v2.11.4"; hash = "..."; };
# Runs in both the vendor fixed-output derivation and the build.
postPatch = ''
cp ${./go.mod} go.mod
cp ${./go.sum} go.sum
mkdir -p plugged private
cp ${./main.go} plugged/main.go
cp -r ${bodyMatcher} private/bodyMatcher
cp -r ${policyEngine} private/policyEngine
cp -r ${ddosMitigator} private/ddosMitigator
cp -r ${souinFork} private/souinFork
chmod -R u+w private
'';
vendorHash = "sha256-...";
subPackages = [ "plugged" ];
}

The derivation addresses several Go module packaging constraints:

  1. Subdirectory entrypoint: The build creates a plugged/main.go file inside a subdirectory rather than at the root of the source tree. Because the root of caddyserver/caddy declares package caddy as a library, placing a package main entrypoint at the root breaks Go module parsing. The entrypoint imports github.com/caddyserver/caddy/v2/cmd to run the standard CLI interface.
  2. Explicit standard modules: The entrypoint must import _ "github.com/caddyserver/caddy/v2/modules/standard".2 Without this explicit blank import, Caddy registers only approximately 45 base modules out of roughly 197 standard modules. Basic directives such as file_server, reverse_proxy, encode, and standard log encoders fail during Caddyfile adaptation.
  3. Relative directory replacement: Private plugins are flake inputs, copied into the source tree at ./private/<input>. The committed go.mod replaces each private module with that relative path:
    Terminal window
    go mod edit \
    -replace github.com/erfianugrah/caddy-policy-engine=./private/policyEngine \
    -replace github.com/erfianugrah/caddy-body-matcher=./private/bodyMatcher \
    -replace github.com/erfianugrah/caddy-ddos-mitigator=./private/ddosMitigator \
    -replace github.com/darkweak/souin=./private/souinFork

A replace pointing at a /nix/store path would end up in vendor/modules.txt, and a fixed-output derivation may not reference store paths. The consequence is that vendorHash covers the private plugin sources, so every plugin bump needs a new hash. 4. No network in the build: Until 2026-11-05 the derivation ran go get and go mod tidy during buildPhase, which needed __noChroot = true; plus nix.settings.sandbox = "relaxed" on the router. The output was then not a function of its inputs, and an outage of the Go module proxy or GitHub failed the router deploy. The network step now lives in update.sh; the build only reads the vendored module set. The same change gave the build its own Go 1.26.10 (a go.overrideAttrs on the pinned 1.26.5), because the old GOTOOLCHAIN=go1.26.10 download is impossible in the sandbox.

The resulting binary packages six public plugins and three private plugins:

PluginSource / PinPurpose
caddy-policy-enginePrivate flake inputL7 rule evaluation, header filtering, honeypots
caddy-body-matcherPrivate flake inputHTTP request body matching and variable extraction
caddy-ddos-mitigatorPrivate flake inputAdaptive behavioral rate limiting and kernel jail management
souinPrivate forkRFC-compliant HTTP cache handler
caddy-dns/rfc2136v1.0.0RFC 2136 DNS-01 dynamic updates to local Knot DNS6
caddy-dns/cloudflarev0.2.4Cloudflare DNS-01 fallback for public hostnames
mholt/caddy-l4v0.1.2Layer 4 TCP proxying (SSH passthrough)
mholt/caddy-dynamicdnsa5890c9Dynamic DNS record synchronization
caddyserver/cache-handlerv0.16.0Standard HTTP cache directive integration
storages/nuts/caddyv0.0.19NutsDB storage backend for cache handlers

Transitive Go dependencies pulled in by plugins can introduce build breakage or security vulnerabilities. The Nix packaging enforces explicit version overrides to maintain reproducible builds and security compliance.

On 2026-09-29, fleet vulnerability audits identified open Dependabot alerts across transitive dependencies in caddy-policy-engine and caddy-ddos-mitigator. update.sh writes explicit -replace directives into the committed go.mod:

Terminal window
go mod edit \
-replace google.golang.org/grpc=google.golang.org/grpc@v1.84.0 \
-replace golang.org/x/crypto=golang.org/x/crypto@v0.57.0 \
-replace golang.org/x/net=golang.org/x/net@v0.60.0
  • gRPC v1.84.0: Replaced v1.82.1 to clear high-severity gRPC transport vulnerabilities resolved in upstream releases 1.82.2 and 1.83.1.5
  • x/crypto v0.57.0: Bumped to satisfy security scanner requirements regarding SSH channel denial-of-service vulnerabilities.
  • x/net v0.60.0: Added 2026-10-22 for CVE-2026-78669.
  • No x/text replace: an earlier v0.39.0 pin was removed on 2026-10-22. The module graph resolves v0.42.0 on its own, so the pin had become a downgrade into CVE-2026-56851.

The Common Expression Language (cel-go) dependency in caddy-policy-engine is deliberately held at version v0.28.1. Upgrading cel-go to v0.29.0 or later breaks compilation of Caddy 2.11.4:

github.com/caddyserver/caddy/v2/modules/caddyhttp
cannot use fn (value of type func(interpreter.InterpretableV2) interpreter.InterpretableV2)
as interpreter.InterpretableDecorator in argument to interpreter.NewCall

In cel-go 0.29, the internal interpreter.NewCall signature was refactored to require InterpretableV2, a breaking API change that Caddy core has not yet adopted.3

Security scanners report advisory GO-2026-6094 against cel-go v0.28.1.4 Analysis of the vulnerability confirms it is non-actionable for this proxy: GO-2026-6094 describes a panic condition when evaluating expressions involving NativeTypes. Caddy’s HTTP expression matchers evaluate standard string, path, and header attributes, and never enable or execute NativeTypes evaluation. Holding at v0.28.1 preserves stable builds without exposing the runtime to vulnerability triggers.

What was removed in the September 2026 redesign

Section titled “What was removed in the September 2026 redesign”

Between 2026-09-23 and 2026-09-26, the edge proxy underwent an audit and architectural simplification. The preceding stack incorporated the OWASP Core Rule Set (CRS v4) via Coraza rules, an automated SecRule translation pipeline, and proof-of-work browser challenges.

A 30-day preflight traffic audit (docs/nuke-snapshot-2026-09-23/) across seven production sites revealed a severe disproportion between complexity and defensive value:

  • 84,373 total policy events were logged across 1,013 unique client IP addresses.
  • 0 actual requests were blocked. The entire CRS rule corpus was operating in detection-only mode because enforcing it caused false positives on legitimate client APIs.
  • 342 imported CRS rules generated thousands of notice-level events (such as rule 9100034 request body inspection and rule 930130 scanner user-agent matches), drowning genuine operational signals in noise.
  • One custom rule: A single exclusion rule skipping false positives for the Atuin shell history sync client accounted for the only critical policy configuration.
  • Maintenance overhead: Maintaining the tools/crs-converter translation pipeline and its 4,566-test regression suite required ongoing debugging, with 93 standing CRS conversion regressions remaining unresolved.

On 2026-09-26, the stack was simplified to eliminate unneeded components:

ComponentStatusRationale
OWASP Core Rule Set (CRS)Removed342 generic rules produced noise; 0 blocks across 84k events
Coraza SecRule engineRetiredReplaced by native Go caddy-policy-engine handler
coraza-caddy forkArchivedFork retired 2026-09-29; archived on Forgejo
tools/crs-converterRemovedEliminated 4,566-test converter test suite and standing regressions
Challenge interstitialRemovedSHA-256 hashcash PoW and service worker tracking removed from edge path
WAF UI settings pagesPrunedRemoved 9 complex configuration views from the dashboard

An initial cleanup attempt on 2026-09-26 removed the entire WAF subsystem. This was immediately corrected: the removal target was CRS and Coraza rule translation, not the underlying policy engine. The in-house caddy-policy-engine was restored to evaluate lightweight, hand-written rules (such as honeypots, path deny-lists, and Atuin exceptions) without the operational burden of third-party signature sets.

edgectl (renamed from wafctl on 2026-09-27) is the companion management daemon running alongside Caddy on the router. It operates as a native systemd service (edgectl.service) listening on 127.0.0.1:8082 (leaving host port 8080 dedicated to Composer).

Caddy runtimeedgectl control planeStorage tier (local sockets)caddy.service(user: caddy)/var/log/caddy/combined-access.logwrites requests/var/lib/caddy/waf/policy-rules.jsonreloads 5s/var/lib/caddy/waf/jail.jsonmtime polledgectl.service(user: edgectl, group: caddy):8082 REST APItails offsetwrites ruleswrites jailStatic Astro/React UI/var/lib/edgectl/uiserves the dashboard hostPostgreSQL 18peer authevents & configValkey socketIP jail cachejail mirrorcountry.mmdbMaxMind GeoIPpure-Go lookup

The control plane interaction handles data persistence and service boundaries:

  1. Permission isolation: edgectl runs under its own system user (edgectl) but belongs to supplementary group caddy. The directory /var/lib/caddy/waf is set to mode 2775 (setgid caddy), allowing edgectl to write configuration files that caddy reads.
  2. Storage transition: On 2026-09-30, edgectl cut over from flat JSON files to PostgreSQL 18 (postgres:///edgectl?host=/run/postgresql) via Unix socket peer authentication. PostgreSQL stores rule definitions, managed lists, and security events. A dedicated local Valkey instance (/run/edgectl-valkey/valkey.sock) mirrors active jail entries.
  3. Decoupled file sync: Caddy does not communicate with PostgreSQL or Valkey directly. edgectl exports compiled rule sets to /var/lib/caddy/waf/policy-rules.json and active IP jail entries to /var/lib/caddy/waf/jail.json. Caddy hot-reloads these files asynchronously. If edgectl or the database stops, Caddy continues serving traffic using the last written state on disk.
  4. Dashboard interface: The static frontend (Astro 7 and React 19) is built by Nix from the same pinned caddy-compose revision as edgectl (pkgs/edgectl-ui.nix, buildNpmPackage), and EDGE_UI_DIR points at its store path. Caddy reverse proxies the LAN-only dashboard hostname to edgectl, which serves both the API and the static files. Before 2026-11-05 the files were copied by hand out of the CI image into /var/lib/edgectl/ui, so a rebuilt router came up without a dashboard and the live copy drifted: it still carried nine pages the September pruning had removed from the source.

The native NixOS deployment carries forward core traffic management, threat intelligence, and logging patterns established in the edge architecture.

Site definitions in the router’s edge/Caddyfile compose modular snippets to enforce traffic tiers:

(ddos) {
ddos_mitigator {
jail_file /var/lib/caddy/waf/jail.json
threshold 0.65
base_penalty 60s
max_penalty 24h
warmup_requests 1000
}
}
(waf) {
import ddos
policy_engine {
rules_file /var/lib/caddy/waf/policy-rules.json
reload_interval 5s
}
handle_errors 400 403 429 {
import error_pages
}
}
(lan_only) {
@denied not remote_ip 10.0.0.0/8 100.64.0.0/10 172.16.0.0/12 192.168.0.0/16
respond @denied 401
}
(research_auth) {
@denied {
not remote_ip 10.0.0.0/8 100.64.0.0/10 172.16.0.0/12 192.168.0.0/16
not header Authorization "Bearer {$RESEARCH_TOKEN}"
}
respond @denied 401
}

The snippets partition services into four access tiers:

  • Public services with WAF (import waf): Evaluates caddy-ddos-mitigator anomaly scoring and caddy-policy-engine rules on internet-facing endpoints such as the Forgejo host and the resolver’s DoH endpoint.
  • LAN and Tailnet restricted (import lan_only): Enforces CIDR filtering allowing only RFC 1918 private subnets and Tailscale CGNAT addresses (100.64.0.0/10). Applied to internal administrative consoles such as the edgectl dashboard and the traffic-analysis UI.
  • Token authenticated (import research_auth): Bypasses authentication for LAN and tailnet clients, but requires a pre-shared bearer token ({$RESEARCH_TOKEN}) for external requests. Applied to compute services such as the search and crawler endpoints of the research stack.
  • Layer 4 SSH passthrough: Caddy uses caddy-l4 to proxy raw TCP traffic for port 2222 (forwarded to the documentation SSH daemon on port 2222) and port 2223 (forwarded to Forgejo’s container SSH server on port 2222).

Malicious IP defense relies on the stamparm/ipsum threat intelligence feed, aggregate data compiled from over 30 public threat lists:7

  1. Daily retrieval: edgectl downloads ipsum.txt daily at 06:00 UTC (configured by EDGE_BLOCKLIST_REFRESH_HOUR=6). Operators can also trigger a refresh on demand from the dashboard.
  2. Score grouping: IPs are parsed and categorised across all eight threat score levels (1 through 8). IPs with a score of 1 or higher are ingested into eight managed lists (ipsum_level_1 through ipsum_level_8).
  3. Rule synchronization: edgectl compiles these lists into policy engine block rules named IPsum Block (Level N).
  4. Execution: The rules are written to policy-rules.json. When a matching IP connects, Caddy’s policy engine blocks the request with a 403 Forbidden before backend evaluation occurs.

For security analytics and geographic rule matching, edgectl resolves client IPs through a priority chain:

Priority 1: Cf-Ipcountry header (0 ms latency, injected by Cloudflare CDN)
Priority 2: Local MMDB lookup (<1 µs, pure-Go reader against /var/lib/edgectl/geoip/country.mmdb)
Priority 3: Online API fallback (cached in memory with an LRU cache, 24h TTL)

The local MMDB lookup uses a pure-Go MaxMind reader requiring no CGO dependencies or external shared libraries. If the Cloudflare header is absent and the local database does not match, the query falls back to a rate-limited online lookup.

Instead of maintaining separate web server and security audit logs, Caddy writes a single unified access stream to /var/log/caddy/combined-access.log. Security metadata is injected using log_append directives in the (site_log) snippet:

(site_log) {
log_append request_id {http.request.uuid}
log_append ddos_action {http.vars.ddos_mitigator.action}
log_append ddos_fingerprint {http.vars.ddos_mitigator.fingerprint}
log_append ddos_z_score {http.vars.ddos_mitigator.z_score}
log_append ddos_spike_mode {http.vars.ddos_mitigator.spike_mode}
log_append policy_action {http.vars.policy_engine.action}
log_append policy_rule {http.vars.policy_engine.rule_name}
log_append policy_ja4 {http.vars.policy_engine.ja4}
log_append policy_score {http.vars.policy_engine.anomaly_score}
log_append policy_detect_rules {http.vars.policy_engine.detect_rules}
log_append policy_detect_matches {http.vars.policy_engine.detect_matches}
log combined {
level INFO
output file /var/log/caddy/combined-access.log {
roll_size 100MiB
roll_keep 10
}
format json { time_format wall }
}
}

edgectl’s AccessLogStore tails /var/log/caddy/combined-access.log using byte offset tracking saved to /var/lib/edgectl/.access-log-offset. When log rotation occurs, the file size drops below the saved offset; edgectl detects the truncation, resets the offset to zero, and resumes streaming without missing events or re-parsing rolled logs.

The DDoS mitigator (http.handlers.ddos_mitigator) runs before the policy engine in Caddy’s handler chain. The Caddyfile snippet in the traffic tiers section places ddos_mitigator outside policy_engine so blocked traffic never reaches downstream WAF processing, saving CPU under load. The plugin header comment states the ordering explicitly: “log_append runs first (outermost), then ddos_mitigator, then policy_engine.”8

The plugin scores every request through three independent layers, then jails IPs whose score crosses a configurable threshold (default 0.75):

Layer 1 - Global rate gate: Per-IP sustained requests per second across all services, measured over a 60-second sliding window using a 64-slot ring buffer of request timestamps.9 When global_rate_threshold is set (default 0, disabled) and the rate exceeds it, the IP is jailed immediately regardless of path diversity. This catches server-saturating floods that spread across many paths.

Layer 2 - Per-service behavioral profiling: An ipTracker keyed on (IP, host) gives each service its own behavioral profile. Path diversity - unique paths divided by total requests - detects targeted low-rate attacks: a normal user browsing 16 pages scores 1.00, while a flood hitting one path repeatedly scores below 0.01. Volume confidence dampens low-request-count profiles, and rate amplification raises the score when a 60-second window rate is high with low diversity.10

Layer 3 - Host diversity exculpation: A global per-IP host count from hostTracker reduces the effective score at jail-decision time: effectiveScore = rawScore / log2(uniqueHosts + 1).11 A real user hitting 8 services gets a roughly 3.17x score reduction; a DDoS targeting one service gets none. The min_host_exculpation setting (default 2) sets a floor below which dampening does not activate, so first-time visitors to a single host are not prematurely excused.

The ServeHTTP method runs six guarded steps on every request before reaching the backend:

  1. Whitelist check - private subnet addresses pass immediately.
  2. Jail and promoted-CIDR check - 64-shard concurrent map, O(1) per request via IsJailed. If jailed and the adaptive stats engine is in spike mode, only 1 in 64 blocked requests is logged to avoid the synchronous JSON log pipeline becoming the bottleneck.12
  3. Host recording and L1 rate check - global per-IP ring buffer rate compared to threshold.
  4. L2 and L3 behavioral scoring - RecordAndScore on the (IP, host) profile, then divide by host diversity factor.
  5. Jail decision - if score exceeds threshold, the IP is added to the jail with exponential backoff (base_penalty * 2^infractions) and up to 25% jitter to prevent synchronised retry storms.13 The CIDR aggregator checks whether enough IPs from the same /24 (IPv4) or /64 (IPv6) have been jailed to promote the entire prefix.
  6. Pass through - next.ServeHTTP for clean requests.

The hot path benchmarks at approximately 90 ns per request with zero heap allocation.14

The plugin is provisioned once per Caddy process and holds all state in memory:

  • ipJail: 64-shard concurrent map (sync.RWMutex per shard) with TTL-based lazy expiry. A package-level registry maps jail file paths to shared jail instances, so the L4 TCP RST handler (mitigator_l4.go) accesses the same jail as the L7 handler without any IPC.15
  • Infraction history: 24-hour TTL keyed by IP. Counts persist across jail terms so repeat offenders escalate penalties even after expiry or manual unjail.16
  • hostTracker: Per-IP host diversity tracked globally across all services. The host count and ring buffer rate are updated atomically on each request.
  • ipTracker: Per-(IP, host) behavioral profiles with LRU eviction. Each profile stores a ring buffer for the 60-second rate, a bounded set of up to 256 unique paths, and HTTP method counts.
  • Count-Min Sketch: Fingerprint frequency tracking across the whole traffic corpus. Counters are halved every decay_interval (default 30 s) so the sketch responds to traffic shifts.
  • adaptiveStats: Welford algorithm plus dual exponential weighted moving averages for spike detection. Below warmup_requests (default 1000), the stats engine suppresses scoring to prevent false positives during cold start.17
  • CIDR aggregator: Tracks jailed IPs per /24 and /64 prefix. When cidr_threshold_v4 (default 5) IPs are jailed from one prefix, the entire prefix is promoted and any subsequent connection from that range is blocked instantly without per-IP lookup.

Four enforcement mechanisms share the same jail, deployed from fastest (kernel) to slowest (userspace):

LayerMechanismConfig flagCapability required
eBPF/XDPXDP_DROP at NIC driver, approximately 10M packets per secondxdp_drop, xdp_ifaceBPF, NET_ADMIN
nftablesKernel ipset drop, approximately 1.5M packets per secondkernel_dropNET_ADMIN
L4 TCPSetLinger(0) sends RST before TLS handshakeVia caddy-l4 appNET_BIND_SERVICE
L7 HTTP403 Forbidden, approximately 50K requests per secondDefault, always activeNone

nftables uses the google/nftables library with auto-reconnection on netlink failure and tear-down of all rules, chains and tables on Caddy shutdown.18 XDP loads a BPF program via cilium/ebpf; the sync uses diff-based updates (only changed entries) rather than full reloads. Both kernel layers log a warning and fall back gracefully if the required capability is missing at Provision time.19

Configuration is set at Provision time via Caddyfile or JSON and does not hot-reload. The jail file is the dynamic surface: the runFileSync goroutine reads and writes jail.json every sync_interval (default 2 s) under flock.20 On read, new entries from edgectl are merged in without removing existing ones. When an entry present in memory is absent from the file and predates the file’s updated_at timestamp, the plugin treats it as a manual unjail: the IP is removed from the jail, the CIDR counter is decremented, and a 60-second grace period prevents immediate re-jail from stale profile data accumulated before the unjail.21

Infraction history is written to the jail on jail.Add and persisted in memory separately from the jail map. It survives expiry and manual unjail so a repeat offender escalates from its last known infraction count rather than starting over.

The policy engine (http.handlers.policy_engine) evaluates user-defined rules through a 7-pass priority pipeline. The order of passes is determined by numeric priority bands: rules within each band are sorted by priority then by rule ID for deterministic tie-breaking.22

The engine implements six Caddy interfaces: Module, Provisioner, Validator, CleanerUpper, MiddlewareHandler (ServeHTTP), and Unmarshaler for Caddyfile parsing.23 A JA4 TLS fingerprinting listener wrapper (caddy.listeners.ja4) computes fingerprints at connection-accept time for use in rule conditions.

PassPriority bandRule typeBehaviour
150-99allowShort-circuit: skip all remaining passes, pass to next handler immediately.
2100-149block, honeypotReturn 403 Forbidden. In detection_only mode, log but do not reject.
3150-199challengeVerify PoW cookie or serve interstitial HTML.
4200-299skipAccumulate skip flags for downstream passes (detect, rate_limit, block, challenge). Non-terminating.
5300-399rate_limitSliding window counter per rule-defined key. Returns 429 on exceed, or logs only in log_only action.
6400-499detectAnomaly scoring (inbound and outbound). If accumulated score exceeds configured threshold at end of loop, return 403.
7500-599response_headerSet, add, remove, or default-set response headers after upstream processing.

The inbound loop (passes 1 through 6) runs before next.ServeHTTP. Pass 7 and outbound detect rules run after the upstream response. The ServeHTTP implementation wraps the response writer to capture status code and body for outbound matching.24

An allow rule that matches still lets rate_limit rules evaluate for allowed traffic, so rate limits are enforced regardless of allow status.25 A skip rule can suppress specific rule IDs, entire passes, or all remaining evaluation altogether via skip_targets.all_remaining.

Rules are loaded from policy-rules.json as a JSON file generated by edgectl. At load time, every rule is compiled into a compiledRule struct: regular expressions are pre-compiled, operator names are resolved to enums, byte-range validation sets are pre-built as 256-bool arrays, challenge difficulty is bounded, and an needsBody flag is set for rules that reference request body fields.26

Hot-reload runs via mtime polling at the configured reload_interval (default 5 s). The checkReload function stats the user rules file, the optional default rules file, and the JTI denylist file. On any mtime change, it opens the file under a single os.Open descriptor, stats the same descriptor to avoid TOCTOU races, and reads and parses the JSON content.27 If compilation fails, the old rules remain in place. If the file is deleted, rules are cleared and rate limit zones are emptied.28

Rate limit counter state is preserved across reloads for zones whose configuration (events and window) has not changed. Zones whose parameters change get fresh counters; removed zones are evicted.29

  • Compiled rules: Stored behind a sync.RWMutex (read-locked during ServeHTTP, write-locked during reload). Rule count, enabled count, and rate-limit zone count are logged at Provision.
  • Rate limit state: 16-shard concurrent maps using fixed-window interpolation (prevCount * (1 - elapsed/window) + currCount), matching the algorithm used by nginx and envoy.30 A background sweep goroutine evicts expired counters every 30 s. Max keys per zone defaults to 100000 to bound memory growth.
  • JA4 registry: A package-level sync.Map keyed by remote address. The listener wrapper writes fingerprints at Accept() time; ServeHTTP reads them. The wrapper uses a 2-second peek deadline because upstream wrappers such as caddy-l4 clear the connection deadline after route matching - without it, a silent or drip-feeding client parked the serial accept loop during the 2026-08-08 edge outage.31
  • Challenge state: A 32-byte HMAC-SHA256 key for cookie signing, a flag indicating whether any challenge rules exist in the loaded config, and an mtime-polled JTI denylist file for retrospective cookie invalidation.32
  • WAF config: Per-service anomaly scoring thresholds, protocol enforcement limits, and disabled detect-category lists, resolved by host at request time.

The challenge system serves an embedded HTML interstitial page when a challenge rule matches and no valid cookie is present. Client-side JavaScript spawns a Web Worker that performs SHA-256 hashcash proof-of-work, collects 13 browser-probe signals (webdriver, CDC markers, WebGL renderer, audio context, speech voices, permissions timing, languages, chrome.runtime, platform, memory, touch, screen), and sends the solution to /.well-known/policy-challenge/verify.33 The server validates the HMAC, PoW hash, and timing, then runs a 6-layer bot scoring pass. A score below 70 issues an HMAC-signed cookie with configurable TTL that lets the client bypass subsequent challenge rules.

Adaptive difficulty maps a pre-solve signal score linearly to a difficulty range configured per rule (min_difficulty to max_difficulty).34 A session tracking service worker and page-level collector inject beacons for cross-navigation correlation. The JTI denylist enables invalidation of specific cookies without rotating the global HMAC key.

Before the main rule loop, enforceProtocolLimits() checks request-level properties against configurable limits: HTTP method allowlist, HTTP version allowlist, maximum argument count, argument name and value lengths, total query string length, and upload size limits.35 Violations produce synthetic detect matches with CRITICAL severity (+5 anomaly score) using their original CRS rule IDs for event correlation. All limits support per-service overrides via WafServiceConfig; a zero value means not enforced.

Unlike the DDoS mitigator which syncs a jail file bidirectionally, the policy engine reads its rules file unidirectionally: edgectl writes, Caddy polls. A failed hot-reload logs the error and keeps the previous rule set. A removed rules file clears all rules and rate limit zones. The JTI denylist file is polled independently on the same ticker so cookie invalidation updates propagate without waiting for a rules change.

The rlState (rate limit state) pointer is set once at Provision and never set to nil, even during Cleanup, because ServeHTTP reads it without holding the rules mutex. Nil-ing it would race with in-flight requests.36

If the JA4 listener wrapper’s Accept() blocks without its 2-second clientHelloPeekTimeout deadline, the serial accept loop stalls and no new TLS connections are accepted. The deadline is cleared after the peek so slow-but-legitimate TLS clients are not affected.

Challenge rules that bind cookies to the JA4 fingerprint (bind_ja4: true) fail when the client’s TLS fingerprint changes between the interstitial solve and the verification POST - for instance, a browser that rotates its cipher suite order. The cookie is rejected and the client is served a fresh interstitial.

Detection-only mode for block rules (detection_only in WAF config) turns 403s into logged events. The rate limiter has its own independent log_only action per zone. The two settings do not interact, so a block rule in detection-only mode still triggers a block_logged action variable while a rate limit rule with action: log_only produces a different variable path.

Decision guide: native service vs containerised stack

Section titled “Decision guide: native service vs containerised stack”

Both deployment models are maintained in the repository workspace. Selecting between native NixOS and Docker Compose depends on host role and network architecture:

PropertyNative NixOS (caddy-edge.nix)Containerised Compose (deploy/edge/)
Host rolePrimary edge router (MS-01)Standalone Docker host or secondary node
Network topologyDirect host binding (CAP_NET_BIND_SERVICE)Docker bridge, host network, or macvlan
Binary deliveryNix derivation with flake input replaceMulti-stage Dockerfile built via xcaddy
Service managerSystemd units managed by NixOSDocker Compose daemon
Storage layerLocal PostgreSQL 18 and Valkey unix socketsDocker volume bind mounts or containerised DB
Operational stateActive production deploymentInactive deployment alternative
Choosing an edge deployment architectureIs the target host the NixOS edge router?Does the environment require Docker-only orchestration?NoNative NixOS Service(caddy.service + edgectl.service)Direct host networking, nix flake buildYes (MS-01)No (NixOS fleet host)Containerised Compose(deploy/edge/compose.yaml)Isolated containers, xcaddy Docker buildYes

The decision flow follows three criteria:

  1. Deploy as a native NixOS service on the MS-01 edge router to avoid container bridge overhead, bind host ports cleanly with systemd capabilities, and manage dependencies through the flake lockfile.
  2. Deploy via Docker Compose on standard Linux distributions or container worker nodes where Nix package management is unavailable.
  3. Keep the Compose configuration updated in deploy/edge/ as a verified operational fallback if router services need isolation during host maintenance.
  1. Caddyserver, “Extending Caddy,” Caddy Docs. https://caddyserver.com/docs/extending-caddy ↩

  2. Caddyserver, “Standard Modules,” Caddy Docs. https://caddyserver.com/docs/modules/ ↩ ↩2

  3. Common Expression Language, “CEL-Go Releases,” GitHub. https://github.com/google/cel-go/releases ↩ ↩2

  4. Go Security Team, “GO-2026-6094: Panics when evaluating expressions involving NativeTypes in github.com/google/cel-go,” Go Vulnerability Database. https://pkg.go.dev/vuln/GO-2026-6094 ↩ ↩2

  5. gRPC Project, “gRPC-Go Releases,” GitHub. https://github.com/grpc/grpc-go/releases ↩ ↩2

  6. P. Vixie et al., “Dynamic Updates in the Domain Name System (DNS UPDATE),” RFC 2136, IETF. https://www.rfc-editor.org/rfc/rfc2136 ↩

  7. Stamparm, “IPsum: Threat Intelligence Feed,” GitHub. https://github.com/stamparm/ipsum ↩

  8. caddy-ddos-mitigator, mitigator.go lines 1-10 (package doc comment: handler ordering log_append, ddos_mitigator, then policy_engine). ↩

  9. caddy-ddos-mitigator, profile.go lines 1-26 (package doc describing three-layer architecture) and lines 62-73 (rateWindowSize, rateWindowNs, ring buffer constants). ↩

  10. caddy-ddos-mitigator, profile.go lines 139-147 (record method updating per-profile request counts, unique paths, and ring buffer on each request). ↩

  11. caddy-ddos-mitigator, mitigator.go lines 478-488 (hostsForScore logic dividing by log2(uniqueHosts + 1) when host count meets MinHostExculpation). ↩

  12. caddy-ddos-mitigator, mitigator.go lines 417-426 (spike-mode deterministic sampling: 1-in-64 blocked requests logged via FNV-32a hash). ↩

  13. caddy-ddos-mitigator, mitigator.go lines 544-557 (calcTTLForInfractions with exponential backoff and up to 25% jitter). ↩

  14. caddy-ddos-mitigator, README.md lines 271-277 (benchmark on AMD Ryzen 7 7800X3D, single-threaded: Jail IsJailed 9 ns/op, CMS Increment 8 ns/op, total hot path approximately 90 ns per request with zero allocation). ↩

  15. caddy-ddos-mitigator, jail.go lines 270-284 (package-level jailRegistry map keyed by jail file path, getOrCreateJail function). ↩

  16. caddy-ddos-mitigator, jail.go lines 190-200 (infraction history with 24-hour TTL, persists across jail terms). ↩

  17. caddy-ddos-mitigator, mitigator.go lines 204-207 (WarmupRequests defaults to 1000, prevents false positives during startup and low-traffic periods). ↩

  18. caddy-ddos-mitigator, nftables.go lines 1-26 (package doc: nftables ipset management, cleanup on shutdown, auto-reconnection on failure). ↩

  19. caddy-ddos-mitigator, mitigator.go lines 245-255 (nftables setup: on NET_ADMIN unavailable, fall back to nftNoop). ↩

  20. caddy-ddos-mitigator, mitigator.go lines 182-184 (SyncInterval default 2 s) and lines 667-760 (runFileSync with flock, bidirectional merge, and unjail detection). ↩

  21. caddy-ddos-mitigator, mitigator.go lines 717-730 (60-second grace period on manual unjail). ↩

  22. caddy-policy-engine, AGENTS.md lines 61-75 (7 rule types evaluated in priority-band order, sorted by priority then rule ID). ↩

  23. caddy-policy-engine, AGENTS.md lines 267-280 (six Caddy interfaces: Module, Provisioner, Validator, CleanerUpper, MiddlewareHandler, Unmarshaler). ↩

  24. caddy-policy-engine, policyengine.go lines 1385-1493 (outbound response wrapping: responseRecorder, ServeHTTP, then evaluateOutbound). ↩

  25. caddy-policy-engine, policyengine.go lines 896-899 (allow-matched guard skips everything except rate_limit rules). ↩

  26. caddy-policy-engine, AGENTS.md lines 159-161 (compiled counterparts pre-processed at load time for O(1) runtime lookups). ↩

  27. caddy-policy-engine, policyengine.go lines 4771-4785 (loadFromFile: open and stat same file descriptor to avoid TOCTOU). ↩

  28. caddy-policy-engine, policyengine.go lines 4708-4720 (checkReload: file removed clears rules and rate limit zones). ↩

  29. caddy-policy-engine, ratelimit.go lines 1-14 (package doc: counter state preserved across hot-reloads, only zones whose config changes get fresh counters). ↩

  30. caddy-policy-engine, ratelimit.go lines 7-11 (package doc: fixed-window interpolation matching nginx/envoy/cloudflare). ↩

  31. caddy-policy-engine, ja4_listener.go lines 1-5 and 60-63 (package doc: 2-second peek deadline; unbounded peek parked serial accept loop during 2026-08-08 outage). ↩

  32. caddy-policy-engine, AGENTS.md lines 220-223 (JTI denylist written by edgectl, polled by plugin for retrospective cookie invalidation). ↩

  33. caddy-policy-engine, AGENTS.md lines 321-327 (6-layer bot scoring with 13 browser probes at L3). ↩

  34. caddy-policy-engine, challenge.go lines 37-47 (ChallengeConfig fields: MinDifficulty, MaxDifficulty for adaptive range). ↩

  35. caddy-policy-engine, AGENTS.md lines 82-103 (CRS protocol enforcement settings and their CRS rule IDs). ↩

  36. caddy-policy-engine, policyengine.go lines 772-777 (Cleanup comment: rlState intentionally not nilled to avoid data race with in-flight ServeHTTP). ↩