Agents and LLMsHarness and toolingreference
Documentation over SSH for coding agents
A coding agent looking up framework documentation has three options: a web search it cannot trust, a documentation MCP server that returns whole pages, or a local clone of the upstream’s docs that nobody keeps fresh. This is a fourth: a server that mirrors the sources as markdown, indexes them, and answers over SSH, because every coding agent already has a shell.
Every fact below is read from the docs-ssh repository at
eed8f7f (2026-10-21; first written against 952a72a, 2026-09-30)
and from the deployment files in it. The runtime half is described from the
code that implements it, apart from the few behaviours checked against the live
server, which the evidence table lists with their dates. The link graph, the
docs_links tool and the persistent query log were merged at that commit and
went live with the 2026-10-22 build. Source counts come from src/application/sources.ts at
that commit with the module imported and its built array counted; the token
figures in the efficiency table are the repository’s own benchmark baselines,
captured 2026-04-07 against third-party documentation MCP servers. The image
layer section was added on 2026-11-10 and is read from
8f2357b.
- The corpus is a plain markdown tree under
/docs/. The server is a Docker image withsshdin front of it and aForceCommandthat routes every session, so the query surface is a file tree andrg, not an API. - Search runs on a pre-built index, not the corpus.
build-index.tswrites one TSV row per markdown file holding path, title and summary, anddocs_searchmatches against that. The repository’s own figure is that the index is about 15 times smaller than the files it describes, which is the reason a search costs hundreds of tokens rather than tens of thousands. - Multi-word queries OR their tokens and rank rows by how many distinct tokens each row hits. An AND chain was the earlier behaviour, and it zero-resulted on any query phrased as a sentence.
- A second transport carries the same seven tools for chat clients that cannot
open a shell: Streamable HTTP at
https://docs.erfi.io/mcp. It is stateless on purpose, and what makes that possible is that the corpus is baked into the image and never written to. - 366 sources at the pinned commit: 183 through a git sparse checkout, 182 over HTTP with 145 of those fetched page by page, 1 over rsync.
- Every fetch also writes a link graph: one row per doc-to-doc link the
markdown already contains, 506,251 of them across the corpus.
docs_linksreads it to answer what a page links to and what links to it. - Ingestion is three passes: a format converter chosen per file, an extension-based fallback, then cleanup. A converter can drop a page outright when its converted form carries no documentation.
- OpenAPI specifications become per-tag markdown at ingestion, one file per tag
plus an
api/overview.mdindex. - Client configuration is generated by the server (
agents,tools) from live container data. The clients that cannot use that path are hand-ported forks, and they drift. - Host keys live on a volume since 2026-10-19, so the key survives a redeploy and a client can pin it. Before that they were generated per container start.
- Queries are logged to a file that survives redeploys for 30 days, recording the operation and its arguments rather than only the shell pipeline that implemented it.
Topology
Section titled “Topology”The same picture as text:
- Three fetching paths supply the tree. 183 sources are cloned by sparse checkout of the upstream’s documentation directory, 182 are fetched over HTTP with 145 of those fetched page by page, and one is mirrored over rsync.
- Ingestion normalises every file through three passes and writes markdown
under
/docs/, plus a per-source stamp recording each file’s origin URL. build-index.tsscans the tree and writes/docs/_index.tsv, one row per markdown file. The origins stamps merge into/docs/_origins.tsv, andlink-graph.tsresolves every markdown link against those origins into/docs/_links.tsv.- The image ships the tree read-only.
sshdon port 2222 routes every session throughlog-cmd, which either runs a builtin or executes the command against/docs/. - The same image runs a second listener on port 8080 carrying the MCP endpoint and a static landing page. Both surfaces read the same tree.
Which path to use
Section titled “Which path to use”| Situation | Path | What it costs |
|---|---|---|
| Coding agent that already has a shell | The seven docs_* tools over SSH, or the generated extension file | One SSH round trip per call; the server caches identical commands |
Agent that can run rg and bat but has no tool file installed | Raw ssh exec against /docs/ | No install at all; the agent writes its own pipelines |
| Chat client in a web interface | https://docs.erfi.io/mcp | No shell needed; HTTP, and the client must support remote MCP |
| You, looking something up by hand | An interactive ssh, or the same rg/bat/tree commands | Nothing. The builtins are conveniences over the tree |
The two transports are not alternatives in the sense of different capabilities. They run the same seven operations with the same output formatting, over two different connections to one tree.
Why SSH
Section titled “Why SSH”The argument for SSH is that the client already has one. A shell command needs
no connector registration, no OAuth flow, and no per-client configuration,
which matters when the same documentation should be reachable from agents on
several machines and in several harnesses. The generated agent instructions are
an ssh invocation with no wrapper around it.
The server constrains what a session can do in return.
sshd_config
sets ForceCommand /usr/local/bin/log-cmd for every session, and log-cmd.sh
then routes to one of three things: an interactive shell logged in as the docs
user, a builtin, or an explicit command string. Authentication is passwordless
by design - PasswordAuthentication yes, PermitEmptyPasswords yes,
PubkeyAuthentication no, AllowUsers docs - because the corpus is public
documentation and there is nothing behind the login to protect. The filesystem
is read-only and everything a client can reach is under /docs/.
The command surface
Section titled “The command surface”log-cmd.sh reads SSH_ORIGINAL_COMMAND, takes its first word, and routes.
Builtins
Section titled “Builtins”help, sources, agents, tools and setup are dispatched to scripts in
commands/. The first two force colour because their output goes to a human
terminal; agents, tools and setup do not, because their output is piped
into a file or read by an agent. tools pi is a sub-dispatch to a separate
script.
sources is dynamic. It walks /docs/ and reports a file count and a size per
source rather than reading a manifest, so nothing in the image carries a
precomputed list that could disagree with the tree.
The seven agent tools
Section titled “The seven agent tools”| Tool | What it reads | Why it is its own operation |
|---|---|---|
docs_search | /docs/_index.tsv | Title and summary only, so it is cheap enough to call first |
docs_summary | The headings of one file | The outline, before committing to a read |
docs_read | One file, whole or a line range | Line ranges keep a large file out of context |
docs_grep | rg inside one file or source | Regex with context, for a phrase the index cannot match |
docs_find | Filenames by glob | When the name is known and the content is not |
docs_sources | /docs/_sources.json | Which sources exist, and how large each one is |
docs_links | /docs/_links.tsv | Pages one link away, in both directions, once the right page is found |
All seven exist in three places that share one output contract: the shell
templates in src/commands/tools-template.ts and tools-pi-template.ts
(generated into commands/tools.sh and commands/tools-pi.sh), and the MCP
server’s src/mcp/docs-service.ts. The shell and MCP versions run the same
rg/bat/find/awk pipelines and differ only in how they reach the tree, so
a query returns the same shape whichever way it arrived.
The generated shell files are tracked in git. pnpm generate:tools regenerates
both, and CI runs git diff --exit-code commands/tools.sh afterwards, which is
what keeps that one in step with its template. The Pi file is regenerated by the
same command but is not diff-checked, so it can drift without failing anything.
Changing the output means editing the template and regenerating, not editing the
shell.
Arbitrary commands and caching
Section titled “Arbitrary commands and caching”Anything that is not a builtin runs as
timeout "$CMD_TIMEOUT" /bin/bash -c "$SSH_ORIGINAL_COMMAND", with a 60-second
ceiling. That is the escape hatch: rg -i 'RLS' /docs/supabase/,
bat /docs/postgres/indexes.md, tree /docs/aws/ -L 2, or rg --json for a
structured consumer.
Identical commands are served from a tmpfs cache. The reasoning is that the docs
are immutable for the container’s lifetime, so the same command always produces
the same bytes and the cache can key on the command string alone. A query the
cache does not hold pays the real cost; a repeat pays nothing. A client may put
a logging marker in front of the command (see the query log below); log-cmd.sh
strips it before the cache key is computed, so two sessions asking the same
thing still share one entry.
Output is capped at 51,200 characters (MAX_RESULT_CHARS in
src/mcp/docs-service.ts and tools-template.ts) rather than truncated
silently. The tail of an over-long result is replaced by a hint naming the file
and suggesting docs_read with offset and lines, so the caller gets a next
step instead of a partial page.
The index, and why search runs on it
Section titled “The index, and why search runs on it”What the index holds
Section titled “What the index holds”build-index.ts
writes one TSV row per markdown file: relative path, title, summary. Titles come
from frontmatter or the first heading; summaries are built from up to five
leading headings, truncated at 200 characters for a title and 300 for a summary,
with tabs and newlines flattened to spaces so a literal frontmatter block cannot
produce a continuation line that a downstream awk reads as another row.
The parser does real YAML parsing for frontmatter, so block scalars and an
embedded --- inside a string value survive, and it tracks code fences under
both delimiters so a heading inside a fence does not become a title. It replaced
an awk implementation that handled a narrow subset and mishandled the rest,
which produced an index whose rows were wrong in ways the search could not
report.
Every *.md under the docs root gets a row, including files whose title could
not be extracted. Those keep an empty title field and stay discoverable by
filename. Files without a row do not appear in search.
Search semantics
Section titled “Search semantics”docs_search runs rg over _index.tsv, not over /docs/.
- Tokens OR, they do not AND. A multi-word query is split and any token may match a row. Rows are then ranked by how many distinct tokens they hit, ties stable on the index’s own order. The AND version required every word verbatim on the same title-and-summary line, which returned nothing for a query phrased as a sentence.
- A miss falls through to the tree, and says so. When the index returns no
rows, the tool searches filenames and then file contents, and prefixes the
result with
[no index matches - found via filename/content search]. The prefix tells the caller the index had no summary match, so the list is a weaker signal than a normal search.
A hit list can also be truncated. Over the requested limit, the result carries
[showing N of M results - refine query or add source filter] rather than
silently dropping rows.
Origin URLs
Section titled “Origin URLs”The index answers where a file is in the mirror. It cannot answer where the page
lives publicly, and an agent citing /docs/supabase/guides/auth.md has cited a
path that exists inside one container.
A second file covers that. /docs/_origins.tsv maps <source>/<relative path>
to the file’s origin URL, built at ingestion time and persisted per source in a
.stamp.json that only a real fetch rewrites, so a cached run keeps it. Then
docs_read and docs_summary prepend it as a [url] line under [source], and
docs_search appends it as a fourth tab-separated column. Both are an awk
lookup guarded by a file-size test, so a mirror built without the file produces
byte-identical output to the version before it existed.
The mapping is per source type. A source cloned from GitHub gets a blob/HEAD
URL, one from GitLab -/blob/HEAD, one from Gitea or Forgejo
src/branch/<ref>, an RFC gets its rfc-editor.org page, and a source whose
published site does not mirror the repository layout gets an explicit mapper:
src/application/public-urls.ts maps this documentation site, ingested as the
erfi-technical-blog source, from src/content/docs/<slug>.mdx to
https://erfi.dev/<slug>/. A converter whose output is not one-to-one with
repository files carries no origin rather than a wrong one.
An llms-full source arrives as one file that the fetcher splits into pages,
and until v0.28.3 every page from such a split cited the source’s root URL:
all 6,215 Cloudflare pages pointed at developers.cloudflare.com/. The dumps
already carry each page’s address, in a different form per publisher: a url:
field in the page’s frontmatter (Next.js), a frontmatter block holding only the
URL before the page heading (Vitest, Vite, Bitwarden), a URL: line under the
heading (Deno), or the page’s own “View as Markdown” link (Cloudflare). The
splitters now read those, and a link to the markdown twin of a page maps to the
HTML page. Measured on the live dumps on 2026-10-21, 7,717 pages gained their
own URL and no file path changed. Neon, Timescale, React Native, Svelte and Porkbun
ship dumps with no per-page address, so they still cite the root.
That column is the difference between a citation a reader can follow and a path that only exists inside a container. It is also why the corpus can cite itself: the mirror holds this site.
The link graph
Section titled “The link graph”Documentation already links to related documentation. The index cannot use
that: it holds one row per file, and a search that finds the right page has no
way to reach the pages next to it. _links.tsv keeps the links as data.
src/application/link-graph.ts runs at the end of every fetch, after
_origins.tsv exists. For each markdown link in a file, outside fenced code and
excluding images, it tries two resolutions in order:
- Through origins. Resolve the link against the page’s own origin URL,
normalise it (scheme set to
https; fragment, query,index.html, extension and trailing slash removed) and look it up in the reverse of_origins.tsv. This is the one that crosses sources: a Supabase page linking to the Postgres manual lands on the mirrored Postgres page. An origin URL shared by three or more files is a source’s root fallback, not a page, and is ignored. - By path, inside the same source. A site-relative or page-relative link is
matched against the source’s own file paths by suffix. The suffix may drop
leading segments but never below two, so
/about/news/installcannot match a file calledinstall.md, and a../that climbs out of the source matches nothing.
Navigation is filtered last: in a source with 20 or more pages, a target linked from more than half of them is a header or sidebar link, and its edges are dropped. A version that also matched absolute links on a source’s own host by path was measured and removed: it added about 2,500 of 506,000 edges, and on sources whose file names are title slugs rather than URL paths those matches could be wrong.
On a copy of the live corpus taken on 2026-10-21 the build writes 506,251 edges, about 17,800 of them between different sources, into a 46 MB file in 41 seconds at a peak of 681 MB of memory. It reads one file at a time and keeps only paths, origins and edges; the first version held every page’s link list at once and peaked at 3,393 MB. The file the live server built on 2026-10-30 held 553,816 edges, 26,425 of them between different sources, in 50,212,693 bytes; the corpus is refetched daily, so these counts move with it.
The query side is docs_links: an rg -F prefilter over the file, then an
awk check that the path matches a whole column, so a page whose path merely
contains the requested one does not match. Rows carry their titles from
_index.tsv, each list is capped at 50 by default, and a lookup on the full
file takes 68 ms. A graph database would answer the same one-hop question; a
sorted TSV and two standard tools answer it without a server, which matches how
every other part of this mirror is queried.
This site has the same structure on a smaller scale. Every page here ends with a list of the pages that link to it and a link graph, built from the same kind of markdown links, and the mirror ingests this site as a source. Run over that source on 2026-10-21, the link graph builder found 557 edges among 154 pages, and for this page it reports the same six outgoing links and two backlinks that its own panel at the foot of the page shows.
How sources get in
Section titled “How sources get in”Source types and the preference order
Section titled “Source types and the preference order”At the pinned commit the built array is 366 sources: 183 git, 182 http, 1
rsync. Formats after normalisation are 193 markdown, 111 HTML, 32 MDX, 24
OpenAPI, 3 text, 2 Go doc, 1 AsciiDoc.
The repository states a preference order over fetch mechanisms, most to least durable, and the counts at the pinned commit follow it:
| Order | Mechanism | Sources | Why it is preferred |
|---|---|---|---|
| 1 | git sparse checkout | 183 | Markdown direct from the source, no rendering |
| 2 | HTTP bulk archive (discovery: "tarball") | 0 | One archive, and survives an upstream redesign |
| 3 | HTTP llms-full | 15 | A single AI-targeted dump, common and stable |
| 4 | HTTP openapi | 22 | A specification is data, not a page to scrape |
| 5 | HTTP page-by-page discovery | 145 | Last resort; brittle to JavaScript rendering and URL changes |
| - | rsync module | 1 | The RFC corpus, the one source with no HTTP or git form |
Two of the 183 git sources (aws-api, gcp-api) also carry OpenAPI: they clone
a directory of specifications and convert it like row four. The 145 in row five
each pick a discovery method, and no ranking is claimed between them:
| Discovery | Sources | Discovery | Sources | |
|---|---|---|---|---|
toc | 49 | dokuwiki | 2 | |
llms-txt | 38 | incidentio | 1 | |
sitemap | 24 | mslearn | 1 | |
mediawiki | 6 | texinfo | 1 | |
sitemap-index | 4 | none (an explicit URL list) | 16 | |
rss | 3 |
A page-by-page source is exposed to upstream JavaScript rendering, URL changes
and rate limiting, and the repository’s notes record what that costs.
cloudflare-blog sits behind Cloudflare’s own bot management, so the default
15-wide page concurrency trips throttling and collapses the source to zero
files, which then misses the 10-minute deadline and lands an empty source in the
image. A fresh CI fetch has no prior baseline, so the regression check does not
catch it. Four concurrent pages with a 40-minute deadline fetch all roughly
3,500 posts with no retries or 429s in about 25 minutes. The shared default is
wrong for one source, and the fix lives on that source.
The three-pass normaliser
Section titled “The three-pass normaliser”Ingestion runs fixed passes in the order the normaliser array in src/index.ts
declares:
- Format conversion, one converter per file.
supportsFormat()picks the converter, so MDX and HTML each get the right one and a file that matches neither is left alone. - Extension-based fallback, for a file whose format the first pass did not claim.
- Cleanup, every remaining normaliser. Markdown cleaning and content
sanitisation run here, because their
supportsFormat()returns false, which is what places them in the last pass.
Three behaviours inside that change what a file becomes rather than how it is formatted:
- A converter may drop a page.
HtmlNormaliserreturns null when the Turndown output is both under 1 KB and under 1 percent of an input over 1 KB. That shape is a single-page-app shell or a paginated listing with no documentation in it, and keeping it would break the invariant that a markdown-capable source contains only markdown files. Dropped pages are logged per source during the fetch. - Markdown arriving as markdown skips conversion. The fetcher sends
Accept: text/markdown, text/html;q=0.9, the negotiation described by the acceptmarkdown specification1 and shipped by Cloudflare as Markdown for Agents2; the underlying media type is RFC 7763’stext/markdown3. When an upstream honours it, the file is tagged as already normalised and pass one is bypassed, because running an HTML-to-markdown converter over existing markdown corrupts it through escaping. Pass three still runs. Two fallbacks guard the edges: a markdown body under 256 bytes is retried as HTML, and a 404 or 406 is retried withAccept: text/html. - One upstream rejects the header outright.
openvpn.netanswers 503 to every page carrying the markdownAcceptvalue and 200 without it, so that source setsskipMarkdownNegotiation. The negotiation is opportunistic per source, and the opt-out is what keeps that from being a wasted fetch.
Sanitisation strips ANSI escapes, null bytes and control characters, and ..
segments are removed from every path at ingestion, so a hostile upstream cannot
write outside its own source directory or put a terminal-escape sequence into an
agent’s context.
OpenAPI specifications to per-tag markdown
Section titled “OpenAPI specifications to per-tag markdown”An OpenAPI specification4 is the one source kind where the input is
already structured, and dumping it as JSON would waste the reader’s budget.
openapi-converter.ts
groups operations by their tag, falling back to the first path segment when an
operation carries none, resolves $ref pointers so a schema arrives inline
rather than as a reference into a component the file does not carry, and emits
one file per tag under api/<tag>.md plus an api/overview.md endpoint index.
Swagger 2.0 and OpenAPI 3.x are both accepted.
That layout is what makes an API surface searchable the same way as prose. The
index rows are per tag, so docs_search returns one endpoint group rather than a
multi-megabyte specification, and docs_read on the file returns exactly that
group.
The RFC corpus over rsync
Section titled “The RFC corpus over rsync”The IETF RFCs are not markdown, not in a git repository, and not behind a
sitemap. They arrive over the RFC Editor’s rsync module5, which the
repository’s notes record is the channel left after the RFC-all tarball was
retired, and which is incremental, so a daily refresh transfers only the files
that changed. Each .txt is wrapped in a fence with an H1 taken from the RFC
header block and the Abstract’s first paragraph as the summary, which gives the
index real titles to match. Around 1 percent of RFCs written before 1990 have
free-form header blocks that defeat that extraction; they get a bare heading and
stay searchable by number.
The MCP endpoint
Section titled “The MCP endpoint”What it exposes
Section titled “What it exposes”https://docs.erfi.io/mcp serves the same seven tools over Streamable HTTP, the
transport in the Model Context Protocol specification revision
2025-11-256. It exists for clients that cannot open a shell: a
chat interface with a custom connector, a desktop client with no terminal. Add
the URL as a remote MCP server and the tools appear; nothing is installed.
The implementation shares the operations with the SSH path rather than
reimplementing them. src/mcp/docs-service.ts holds the seven pipelines with a
configurable docs root and a node:child_process runner where the shell version
has an SSH hop, and src/mcp/server.ts wraps each one in a registerTool with a
Zod schema. The Runner is injectable, so the pipelines are unit-tested without
a real rg or bat.
Statelessness as a property of the corpus
Section titled “Statelessness as a property of the corpus”With no session state and no ordering, there is no session store to lose, instances can sit behind a load balancer with no affinity, and responses can be cached by a CDN.
Cross-origin protection is on (enableDnsRebindingProtection), with Host and
Origin allowlists defaulting to the production hostname, the Fly hostname and
localhost for hosts, and the Claude and ChatGPT web origins. Configuration is
environment only: DOCS_ROOT, MCP_PORT, MCP_HOST, MCP_STATIC_DIR,
MCP_ALLOWED_HOSTS, MCP_ALLOWED_ORIGINS, VERSION and DOCS_LOG_FILE. The
last one is the query log below; appending a line to it per tool call is the
only write the endpoint makes, and no request reads it back.
One listener carries both surfaces on port 8080: POST /mcp for the endpoint, a
static landing page for every other GET, and GET /healthz for the platform
health check. Serving the landing page from the same process is what avoids the
one-service-per-external-port limit on Fly, which is where the design was
written and is still one of the hostnames the allowlist accepts. The server
compiles to one musl binary via bun build --compile, so the runtime image
carries no Node and no node_modules; the Alpine base needs libstdc++ and
libgcc for it. If the binary is absent, the entrypoint falls back to a busybox
HTTP server for the landing page alone, which is how an older image built before
the endpoint existed still serves something.
The query log
Section titled “The query log”Until 2026-10-21 the log was an 8 MB tmpfs, wiped by every redeploy, and most
of its lines were the container healthcheck running echo ok every 30
seconds (125 of the 196 commands logged on the day it was replaced). The rest
were shell pipelines: to learn which page an agent read, you
had to parse the awk and bat command the tool template happened to
generate.
It now lives on a volume (DOCS_LOG_DIR). The entrypoint archives the previous
file under its start time on every container start, which with the daily
redeploy means about one file per day, and deletes archives older than 30
days. A single file stops growing at 50 MB. The healthcheck is not logged; the
echo ok that the deploy workflow sends over SSH to wait for the new container
is, as an ordinary command.
To log operations rather than pipelines, a client prefixes each command with a
shell no-op: : docsop <base64 JSON>; <command>. log-cmd.sh splits the
marker off, runs and caches the bare command, and writes the decoded fields
(op, q, path, source, pattern, direction, sid) into the log line.
A marker that is not plain base64 is discarded. A server that predates the
marker runs it as the no-op it is, so clients could send it before the server
understood it. The Pi and Claude Code clients send it with one random sid per
process; MCP tool calls write the same fields, except sid, from their
arguments.
The session id matters because every SSH session reaches the container through the edge proxy, so every client has the same address, and each tool call is a separate connection. Without a session id there is no way to tell that a search and the read that followed it came from the same agent. With it, the log turns into pairs of a query and the page that answered it, which is the test set any change to search ranking needs before it can be judged.
Generated client configuration
Section titled “Generated client configuration”The agents generator
Section titled “The agents generator”ssh -p 2222 docs@docs.erfi.io agents prints agent instructions, and the
formats are pi, opencode, claude, cursor, gemini, skill (with YAML
frontmatter) and a default that describes the raw SSH patterns for any agent.
The output is built at request time from live container data: the source list
and the file counts are read from /docs/, so an instruction file pulled after
a deploy names the sources that are present rather than the ones that were
present when it was written. The same generator emits a “Related source groups”
section from _source_groups.json, which is derived from the per-source tag
table, so an untagged source does not appear in it.
tools and tools pi are the other direction. They print a client file - a Zod
schema file for OpenCode, a TypeBox extension for Pi - that wraps the same SSH
commands. Nothing executes at install time; the output is a file the operator
reads and saves. setup prints a structured guide an agent can follow to choose
between those paths.
Clients outside the repository
Section titled “Clients outside the repository”This repository’s clients are generated. The clients outside it are not.
The Pi extension and the Claude Code MCP toolkit used alongside it run from a
hand-maintained port of the same tools in a separate dotfiles repository, and
the repository states the consequence plainly: a change to the server’s output
format does not reach them until someone ports it. When origin URLs were added
to the docs_read header, that port was a separate change made later, and until
it landed the two clients returned different headers for the same operation.
docs_links went the same way: the server change and the client port were two
commits in two repositories, made by hand, on the same day.
That is the general shape of the problem rather than a detail of this server. A generated client cannot lag its server; a forked one can, and nothing reports it. Prefer generation where the client can be generated - and when it cannot, treat the copy as a downstream release with its own change to make.
Deploy and release
Section titled “Deploy and release”One workflow,
.forgejo/workflows/build.yml,
covers tag releases, the 02:00 UTC daily refresh and manual runs, all inside one
concurrency group so two builds cannot race for the same image tag.
The plan step
Section titled “The plan step”The workflow does not decide whether to fetch by looking at the calendar. A plan
step (src/ci/plan-cli.ts over planBuild in src/ci/build-plan.ts, both
unit-tested) reads docs/_build.json from the persistent cache, which records
when the docs were fetched and at which commit, and then:
- A tag build always builds, and refetches only if the docs are 20 hours or older, or the fetcher code changed since that commit.
- A scheduled run skips the whole job when the docs are under 20 hours old and the fetcher code is unchanged. A release that just fetched them turns the next daily run into a no-op.
- A manual run with
refresh: trueforces a full refetch.
When the fetcher itself changed, the per-source trust window drops to zero so every source is regenerated with the new code. Otherwise a warm run trusts sources fetched within 36 hours. The expensive step is skipped for a checkable reason rather than on a timer.
Verification gates tags, not the refresh. The verify job runs the tools-file
sync check, typecheck, unit tests and the Docker end-to-end tests; a scheduled
build ships main, which CI has already gated.
The cache
Section titled “The cache”The cache is a Docker volume on the router’s daemon holding the fetched docs/
tree and the clone work directory, moved in and out with docker cp through a
throwaway container so the runner needs no volume configuration. It is saved
right after a fetch and before the image build, so a failed build does not cost
the fetch. Deleting it means the next run is a cold fetch measured in hours
against a three-hour runner job ceiling, which is the failure to avoid.
Image layers
Section titled “Image layers”The corpus is baked into the image, so most of the image is documentation: the
/docs tree is 1.17 GB on disk and 520 MB compressed in the registry. Until
v0.32.0 the runtime stage copied it with one COPY --from=fetcher /docs /docs,
which made it one layer. Registries and the Docker daemon cache layers by
digest, so a change to any file gave that whole layer a new digest, and every
build pushed the 520 MB again and every deploy pulled it again, however few
sources had changed.
Since v0.32.0 a script in the fetcher stage,
src/commands/layer-buckets.ts,
splits the tree before the runtime stage copies it. Each source directory goes
into one of 32 buckets chosen by an FNV-1a hash of the source name, so adding
or removing a source changes only its own bucket. The top-level files
(_index.tsv and the others), which change whenever any document does, go into
a separate layer that is copied last. The runtime stage then has one COPY
per bucket, and a unit test fails if that list and the bucket count disagree.
Splitting alone would not make the layers reusable. A layer records each file’s
modification time, and the fetcher rewrites every file on each run, so an
unchanged source would still produce a new digest. The script sets every
modification time to the Unix epoch; nothing at runtime reads them, and stat
on any document in the container shows 1970-01-01. Two local builds that
differed in one file of one source produced 48 layers each, and only the
bucket holding that source, the top-level layer and the few kilobyte-sized
layers after it had different digests. The other 31 bucket layers matched.
The script has one build-specific fallback. Inside a Docker build the tree
comes from an earlier RUN, so overlayfs holds it in a lower layer, and
overlayfs refuses to rename a directory from a lower layer with EXDEV. The
script copies and deletes when a rename fails that way.
Source health
Section titled “Source health”A separate workflow probes every source daily at 00:30 UTC, before the 02:00
build, and on a push that changes the fetcher, where it probes only the sources
whose definitions changed. A git source is cloned without checkout and its
paths listed; an llms-full or OpenAPI source is downloaded, split and
counted; a source fetched page by page has its discovery run and three sample
pages fetched; the rsync source lists its module. It runs strict, so a degraded source fails
it as surely as a dead one. The rule for an upstream error is narrower than the
rule for drift. A 429 or 5xx on a sample is retried once; a 5xx that survives
the retry is noted but not counted, as is a 429 from the Wayback Machine, which
four sources pin captures on, unless no sample fetched at all. That rule
came from two runs on 2026-10-21 in which ubuntu.com returned 500 for one
sample page each, a different page each time, while the nightly fetch, which
retries with backoff, got all 255 pages.
Reaching the router from a CI job
Section titled “Reaching the router from a CI job”The deploy step posts to Composer and polls the job; a smoke job then connects to the live server. Both need a job container to reach services on the router, and a Forgejo job container cannot reach the router by its public address. The fix was infrastructure-side: split-horizon records that resolve the service hostnames to the router’s service address, and firewall rules that accept the Docker bridges on the SSH and HTTPS ports. The release that revealed the gap failed at the deploy step with a connection timeout after the image push had already succeeded, and its image was deployed by hand.
That belongs to the GitOps deploy path rather than to this server, and it is written up as a runbook in Wiring a Forgejo push to a Composer GitOps deploy, including the command that reproduces the failure. A successful image push leaves the image in the registry, and the GitOps deploy path must still deploy it before the release is finished.
Versioning is a git tag. pnpm release:patch (or minor, major) bumps
package.json, commits, tags and pushes in one command, and the landing page
reads its version from the tag at image build time.
Security posture
Section titled “Security posture”The container is built to have as little to attack as possible. The corpus is public by design, so the threat model is the container rather than the content.
| Control | Setting |
|---|---|
| Root filesystem | read_only: true; /tmp and /var/log are tmpfs; /run/sshd (host keys) and the query log directory are named volumes |
| Capabilities | cap_drop: ALL, then CHOWN, SETUID, SETGID, SYS_CHROOT and AUDIT_WRITE added back, which is what ssh-keygen and sshd need |
| Privilege escalation | no-new-privileges: true |
| Resources | 2 CPU and 512 MB |
| Authentication | Passwordless, AllowUsers docs, PermitRootLogin no, public key auth disabled |
| Key exchange | sntrup761x25519-sha512 first, a hybrid post-quantum exchange, with curve25519 as fallback |
| Ciphers and MACs | chacha20-poly1305 and AES-GCM only; encrypt-then-MAC only |
| Command routing | ForceCommand on every session |
| Audit | Every command and MCP tool call appended to a JSONL log owned by root and group-writable by the docs user, so the session user can append but cannot truncate; kept 30 days |
| Ingest | ANSI and control bytes stripped, .. removed from paths |
| Output | 51,200-character cap, with a truncation hint naming the file |
Reading the numbers
Section titled “Reading the numbers”The repository publishes a token comparison between these tools and third-party documentation MCP servers. Quoted as the repository has it:
| Approach | Tokens | Against the baseline |
|---|---|---|
docs_search, returning file paths | ~480 | 98 percent smaller |
docs_summary, returning headings | ~200 | not comparable |
docs_grep, returning matches | ~1,500 | 77 percent smaller |
| A documentation MCP server returning full pages | ~30,000 | baseline |
1,500 tokens against the ~30,000-token baseline is a 95 percent cut, so the 77 percent in the docs_grep row implies a baseline near 6,500 tokens rather than the 30,000 sitting in the same column.
Read that table for its shape rather than its precision. The MCP baselines were captured on 2026-04-07 from real calls to vendor servers, so they describe those servers on that day and not any server today. The mechanism behind the gap is the index: a search returns paths and one-line summaries, a summary call returns headings, and a read returns only the range that matters. Three small responses replace one large one.
What generalises
Section titled “What generalises”- Publish an index, and search the index. A search never opens a documentation file. Any service answering lookup questions for an agent can make the same trade, and the index is cheap because it is derived from the files it describes.
- Keep the transport separate from the operations. One implementation of the seven operations, two ways to reach it. The alternative, an HTTP server that reimplements what the shell scripts already do, is two things to keep in step.
- Immutability is what allows a stateless endpoint. All seven operations are idempotent reads because the corpus never changes inside a container. That property belongs to the deployment; without it the stateless endpoint would need a session store.
- Generate the client, or expect it to lag. A generated client file cannot disagree with its server. A hand-ported one can, and no test notices.
- Keep relationships as data the existing tools can read. The link graph
is a sorted TSV queried with
rgandawk, the same as the index. One-hop questions do not need a graph database, and a file built at fetch time is immutable for the container’s lifetime like everything else in the tree. - Log the operation, not the implementation. A log of shell pipelines answers nothing about how the tools are used. A marker that carries the operation and a session id costs one no-op per command and turns the log into an evaluation set.
- Make the cache decision explicit. The plan step exists because fetching every time wastes hours and never fetching serves stale documentation, and the question of which is right has a checkable answer: the age of the cached tree and whether the fetcher changed.
Reproducing these figures
Section titled “Reproducing these figures”| Claim | How it was checked |
|---|---|
| 366 sources, split 183 git / 182 http / 1 rsync | Read from src/application/sources.ts at eed8f7f, imported and the built array counted |
| Post-normalisation format split | Same import, grouped by the format field |
| Discovery method counts | Same import, grouped by the discovery field |
| Index row shape, 200 and 300 character caps, five headings | Read from src/commands/build-index.ts |
| Three passes and the drop rule | Read from src/index.ts and the normaliser sources |
| Seven tools, the 60-second ceiling, tmpfs caching | Read from log-cmd.sh, entrypoint.sh and src/commands/tools-template.ts |
| 7,717 llms-full pages gaining their own URL | The splitters at a830ddf run over the ten llms-full dumps fetched on 2026-10-21, before/after comparison of file paths and origins |
| The link graph builder: 506,251 edges, about 17,800 cross-source, 46 MB, 41 s, 681 MB peak (3,393 MB in the first version) | writeLinksTsv at 5182f67 run over a copy of the live corpus on 2026-10-21 |
| Re-count on 2026-10-30: 553,816 edges, 26,425 cross-source, 50,212,693 bytes | wc -l and ls -l on /docs/_links.tsv in the live container; cross-source counted as rows whose two paths start with different source directories |
| 557 edges among 154 pages of this site; six out and two in for this page, matching its panel | writeLinksTsv at eed8f7f over the erfi-technical-blog source and its origins, copied from the live container on 2026-10-21, against the panel on erfi.dev the same day |
| About 2,500 edges from same-host path matching | The same run with and without that rule |
68 ms docs_links lookup | DocsService.links on postgres/sql-select.md against that 46 MB file |
/docs as 32 bucket layers plus a top layer; only the changed bucket gets a new digest | Two local builds of 8f2357b on 2026-11-10 from a small test tree, the second with one file changed and every file’s modification time refreshed; RootFS.Layers compared layer by layer |
| 1.17 GB tree, 520 MB compressed single layer before v0.32.0 | docker history on the deployed image and the amd64 manifest on ghcr, 2026-11-10 |
| Query log retention, size cap, marker fields, healthcheck skip | Read from entrypoint.sh and commands/lib/log.sh; covered by unit tests and a Docker end-to-end test that restarts the container |
| The two ubuntu.com 500s | Check sources runs 82 and 85 on 2026-10-21 |
| 51,200-character output cap | MAX_RESULT_CHARS in src/mcp/docs-service.ts |
| Stateless MCP, JSON responses, origin allowlists | Read from src/mcp/server.ts |
| Container hardening and resource limits | Read from the deployment compose file |
| The token comparison table | The repository’s README.md, whose figures trace to tests/benchmark/token-efficiency.test.ts |
Checked live on 2026-10-21: the per-page [url] headers from v0.28.3, read
back through docs_read for a Cloudflare and a Vitest page. Checked live on
2026-10-30: docs_links on postgres/sql-select.md returned 12 outgoing links
and 26 backlinks; the query log sat on its volume with one archive per container
start since 2026-10-23, and a docs_links call from Pi logged its op, path
and sid fields; an MCP tools/call logged a type:mcp line with the
client’s own address. The other runtime statements about the link graph and the
query log (retention, size cap, marker handling) are read from the code and its
tests. The per-source file counts a live sources call would
report are not stated here.
Related docs
Section titled “Related docs”- Wiring a Forgejo push to a Composer GitOps deploy carries the CI reachability failure that broke this server’s v0.28.0 release, and the fix.
- Composer: a GitOps control plane for Docker Compose stacks is the deploy path this server’s build workflow posts to.
- The Forgejo Actions runner: build and lifecycle explains the job containers the build and smoke steps run in, including the network boundary above.
- A cross-client memory store for coding agents solves the retrieval half of the same problem in the other direction: finding one message among 640k, and why an approximate index has to be measured against exact search before it is trusted.
- A kanban board for coding agents is a third agent-facing service here, and the one that also puts an HTTP surface in front of a database-backed service.
- Keeping credentials out of coding-agent transcripts covers the client-side half of reading untrusted text into an agent’s context.
References
Section titled “References”-
acceptmarkdown.com, “Serve Markdown to AI Agents with Accept Headers.” https://acceptmarkdown.com/ ↩
-
Cloudflare, “Markdown for Agents,” Cloudflare Fundamentals docs. https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ ↩
-
IETF, “The text/markdown Media Type,” RFC 7763. https://www.rfc-editor.org/rfc/rfc7763.html ↩
-
OpenAPI Initiative, “OpenAPI Specification.” https://spec.openapis.org/oas/latest.html ↩
-
RFC Editor, “Download RFCs,” RFC Editor. https://www.rfc-editor.org/retrieve/rsync/ ↩
-
Model Context Protocol, “Transports,” Specification revision 2025-11-25. https://modelcontextprotocol.io/specification/2025-11-25/basic/transports ↩