Three model families reviewed an infrastructure change: what they found and what they missed

The change under review moved a home gateway’s CI onto a second Docker daemon with bounded storage, after a full disk took the gateway’s DHCP server down; the incident and design are on A full disk took DHCP down. It spanned seven repositories: a NixOS module and firewall rules, the forge’s compose file, Go health checks, two CI workflows, Prometheus alert rules, and the public page itself. It had shipped, passed a VM rehearsal, and survived one production regression by the time it was reviewed.

Three reviewers from three model families read it blind, and a fourth model, the one that wrote the change, judged their findings against the running system. This page records what came back, how each finding was checked, and the regression that none of the three reviewers raised.

Provenance

One review, run 2026-11-12, n=1. Counts come from the review record; “verified” means the lead ran a probe on the live gateway, read the cited source code, or queried the forge’s CI task history in the same session. Nothing here is a benchmark of the models in general.

The setup

Role	Model	Saw
Reviewer A	DeepSeek V4 Pro	The diffs, the plan, the public page, a written contract
Reviewer B	Kimi K3.1 Pro	Same
Reviewer C	Gemini 4 Argon	Same
Lead	Claude Opus 5.5 (the change’s author)	Everything, plus shell access to the gateway and the forge

The reviewers got a contract, not the author’s reasoning: what the change had to guarantee (CI cannot use disk the gateway’s services need; the main Docker daemon, DHCP, DNS and existing firewall behaviour stay unchanged; CI keeps reaching what it reached before and nothing more; boot never blocks; alerts fire before the disk fills; the public page matches the code and leaks nothing private). They were read-only and could open any file in the seven repositories. The usual second, informed pass, which shows each reviewer the author’s intent, was skipped; the lead checked the load-bearing claims directly instead.

The numbers

	A	B	C	Total
Findings	9	9	10	28
Labelled critical	0	2	4	6

After merging duplicates and checking each claim, the lead sorted them into 10 items to act on, 3 to consider, and the rest dismissed or noted. The ten were all confirmed real; the dismissed ones were hypothetical, pre-existing, or wrong.

Where the families overlapped:

Problem	Raised by	How it was checked
Published ports did not work where tests need them	A, B, C	Read the firewall rules against where the daemon binds ports
Host services exposed to CI on the CI bridge addresses	B, C	Probe from a CI container: four host ports open, including two management APIs
Forward rules matched on source address only, so a spoofed address from the IoT VLAN got one-way internet egress	B, C	Read the rule set; reverse-path filtering is on for the WAN interface only
A free-space gate in one workflow sized for half its real peak	B, C	Walked the workflow’s copy steps
A health check that reports OK when its own probe fails	B, C	Read the Go code: four error branches return OK

Two findings came from one family only and were both real: the runner holding a dead Docker socket after any restart of the CI daemon (C), and CI’s runner cache and the Windows build VM’s disk still living on the root filesystem, which made the public page’s containment claim false (B). The socket finding was the most serious item in the review. It is easy to check once stated: the daemon removes and recreates its socket file on every start,¹ and a file bind-mount keeps pointing at the old one.

What the checking caught

Checking each claim against the system mattered in both directions.

• A claim that job-created networks had no DNS (C) was tested from a container on such a network: DNS worked, because Docker’s embedded resolver answers it. Dismissed.
• A claim that a cleanup script aborts when one removal fails (A) misread set -e: bash does not exit on a failing command inside an && list unless it is the last one.² Dismissed.
• A claim that an alert’s guard made it page constantly (C) had the arithmetic wrong: with about 200 GB available, the trend alert needs roughly 8 GB an hour sustained for six hours. Dismissed.
• The exposed-ports finding was raised as a hypothesis by two reviewers and confirmed by a probe: four host ports answered from a CI container. It stayed an act-on item. Its severity was lowered on the evidence: CI jobs already hold a Docker socket and run privileged, so they could reach those ports anyway.

What none of them found

The most urgent problem in the change was not in the review. Since the switch, every job that cloned a private repository over the forge’s in-network SSH port had failed: the firewall allowlist for CI named the forge’s HTTP port, the runner’s cache proxy and the Windows VM’s SSH, but not the forge’s own SSH port. Three repositories’ workflows use it.

It came out of one query against the forge’s task history while a different finding was being checked: one repository’s parity job had passed 24 of 24 runs in the two days before the switch and failed every run after it. The job log said ssh: connect to host forgejo port 2222: Connection timed out. A release in another repository was already queued and would have failed the same way. The fix was one port in two rules, deployed within minutes.

The reviewers could not have found it from what they were given. The diffs showed which endpoints CI was allowed to reach. The workflows that needed a fourth endpoint were in repositories outside the review’s scope. Neither the contract nor the plan said which endpoints CI actually used, only which ones the author believed it used. The author’s own VM rehearsal had the same gap: it tested the three allowed endpoints and the one blocked port it was told about.

Afterwards

All ten act-on items were fixed the same day. The same VM rehearsal, extended with a check for each item (over 80 in total, including filling the CI disk and confirming the root filesystem did not move), failed 11 checks on the configuration the reviewers saw and passed every check on the fixed one. On the live gateway, a repository whose integration tests start Postgres through testcontainers ran green on the new port rules, and the Windows build VM booted from its new disk and answered SSH from a CI container.

What to take from it

• Read the system’s own history before or alongside the diff. Which jobs, requests or units started failing after the change is a fact about the running system. Reviewers reading the change can only infer it, and only when the consumers are in scope. Here that was one database query.
• Treat reviewer output as hypotheses. Each of the ten act-on items was confirmed by a probe, a source read or a rule walk, and three plausible findings fell apart the same way.
• Diversity paid. One family produced most of the serious findings, another found the false public claim, and the third added little that survived. Two of the three were needed for full coverage. With one review, that says nothing general about the models.
• A blind contract helps, but it can only test what the contract states. “Keep CI reaching what it reached before” was in it; what CI reached before was not written down anywhere the reviewers could see.

Related: A PCB as code, and proving its checks work records a five-reviewer review of a hardware repository with a similar outcome, findings on top of green gates and one rejected claim that repeated a stale doc. Which model runs which agent work covers how the harness picks models in the first place.

References

1. Docker, go-connections/sockets/unix_socket.go, NewUnixSocketWithOpts (unlinks the path before listening). https://github.com/docker/go-connections/blob/main/sockets/unix_socket.go ↩

2. Free Software Foundation, “The Set Builtin,” GNU Bash Reference Manual. https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin ↩

Linked from

• A full disk took DHCP down: isolating CI storage on a gateway

Link graph: 1 doc links here, this page links to 3