Forge and CIreference
Vigil fleet vulnerability sweep, exploitability enrichment, and CI gating
Scanning a homelab container fleet with raw vulnerability scanners produces long lists of unranked CVE rows, most of them unfixable Debian or Alpine base-layer advisories on services that are not reachable from the internet. vigil is a single-binary Go CLI (Go, with gopkg.in/yaml.v3 as its sole external dependency) that orchestrates Trivy1 across Docker Compose trees, enriches every CVE with CISA Known Exploited Vulnerabilities (KEV)2 membership and FIRST Exploit Prediction Scoring System (EPSS)3 probabilities, joins each image onto a static exposure registry (stacks.yaml), and diffs consecutive sweeps so the weekly digest highlights what changed and which findings are both reachable and exploitable.
The vigil CLI and weekly router sweep pipeline launched on 2026-08-20; its CI and multi-architecture container build workflows moved from GitHub Actions to the self-hosted Forgejo instance on 2026-11-16, where vigil also opted into the self-hosted Renovate sweep for Dockerfile and Go module updates.
- Trivy for CVE matching,
vigilfor enrichment and ranking:vigildelegates package and OS advisory matching totrivy image --scanners vuln1 (ExecScanneror offlineDirScanner), adding exploitability context, reachability ranking, and state diffs on top of Trivy’s JSON output. - Runtime exploitability feeds:
vigil scanfetches the CISA KEV catalog2 and queries the FIRST EPSS v1 API3 in batches of 100 CVEs at run time. Each feed degrades on its own: if KEV is unreachable, KEV flags stay false; if EPSS is unreachable, scores stay zero. The sweep logs a warning and still writes the state file. unknownexposure sorts first:stacks.yamlclassifies stacks aspublic(rank1),lan(rank2), orinternal(rank3). Any stack missing fromstacks.yamlreceives tierunknown(rank0). The report orders images by KEV hit count first, then tier rank, so among images with equal KEV counts the unclassified ones sort abovepublicand force triage instead of sinking to the bottom.--ignore-unfixedby default in CI gates:vigil gatefails a build only onCRITICALorHIGHvulnerabilities that carry a non-emptyFixedVersion, preventing unfixable base-image advisories from training operators to bypass the gate.- File state and set-difference deltas: each sweep writes
.vigil/state.jsonatomically via a.tmprename, andvigil report --prevcomputes the set difference on(image, vulnerability ID, package)tuples so weekly reviews start from newly appeared and newly resolved findings. - Paired with Renovate: self-hosted Renovate opens pull requests when upstream tags move;
vigilshows which images carry actively exploited or high-EPSS vulnerabilities, how exposed they are and which fixed versions exist, so the operator knows which image update or base-image rebuild to take first. It does not read Renovate’s pull requests itself.
Architecture and pipeline flow
Section titled “Architecture and pipeline flow”The codebase is split into a thin CLI entry point (cmd/vigil/main.go) and six internal packages with no circular dependencies:
| Package / path | Responsibility | Key types and entry points |
|---|---|---|
cmd/vigil/main.go | Subcommand dispatch (scan, report, gate, healthcheck) and signal handling (SIGINT, SIGTERM) | cmdScan, cmdReport, cmdGate |
internal/inventory | Walks a directory tree for Compose files, resolves ${VAR:-default} expressions, merges --images-file, and attributes each image to its consuming stacks | Image, Collect(root, imagesFile) |
internal/trivy | Runs trivy image (ExecScanner) or reads cached JSON files (DirScanner), parses Results[].Vulnerabilities, and ranks severities | Report, Vuln, Scanner, ExecScanner, DirScanner, SanitizeName |
internal/enrich | Fetches the CISA KEV catalog and batch-queries the FIRST EPSS API (100 CVEs per request, 30s HTTP timeout, 64 MiB body cap) | LoadKEV, LoadEPSS, KEVURL, EPSSURL |
internal/tier | Parses stacks.yaml and resolves the effective exposure tier across all stacks consuming an image | Tier, Registry, Load, ForStacks |
internal/store | Serialises sweep state atomically (.tmp write + os.Rename) and computes New and Resolved deltas between two snapshots | State, ImageResult, Vuln, FindingRef, Delta, Diff |
internal/report | Sorts images and KEV findings and renders the Markdown digest or formatted JSON | SortImages, Markdown(state, delta), JSON(state) |
Why orchestrate Trivy instead of reading raw scanner output
Section titled “Why orchestrate Trivy instead of reading raw scanner output”| Question during triage or CI | Raw trivy image output | vigil (scan, report, gate) |
|---|---|---|
| Which reported CVEs are actively exploited in the wild? | Not distinguished from theoretical CVEs with the same CVSS severity | Annotates every CVE-* with CISA KEV membership2 and FIRST EPSS 30-day exploitation probability3, rendering KEV hits in a dedicated top table |
| Is the affected container reachable from the internet or isolated on an internal bridge? | Scanner only sees the image reference, not how stacks deploy it | Joins every image against stacks.yaml (unknown, public, lan, internal) and promotes shared images to their most-exposed consumer’s tier |
| What changed since last week’s sweep? | Requires diffing per-image outputs by hand | store.Diff diffs .vigil/state.json against --prev on image|id|pkg keys; the report lists new findings and counts resolved ones |
| Should this CVE fail a CI image build? | Failing on all CRITICAL,HIGH blocks deploys on unpatched Debian/Alpine base layers | vigil gate defaults to --severity CRITICAL,HIGH --ignore-unfixed=true, failing only when FixedVersion != "" |
| What happens when a new Compose stack is added to the host? | Unnoticed unless added to a scan list | Directory walk discovers it automatically; if omitted from stacks.yaml, it receives tier unknown and sorts first in the report |
Image discovery and stack attribution (internal/inventory)
Section titled “Image discovery and stack attribution (internal/inventory)”inventory.Collect(root, imagesFile) builds the deduplicated list of container images to scan from two sources: a recursive walk of --root and an optional newline-separated --images-file.
During the directory walk, Collect skips .git, node_modules, .vigil, backup, and backups directories and parses any file matching ^(docker-)?compose(-[\w.]+)?\.ya?ml$. For each service’s image: string, resolve() performs three normalisation steps:
- Strips inline comments beginning with
" #". - Expands
${VAR:-default}variable expressions to their inlinedefaultstring using the regex\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-([^}]*))?\}. - Rejects any reference that still contains an unresolved
${VAR}(no default value) or ends with:local(host-built images that do not exist in a registry).
To attribute a Compose file to a stack name, stackName(path, root) walks upward from the Compose file’s directory toward root until it finds a directory containing a .git/ subdirectory, returning that repository root’s basename (or the immediate parent directory’s basename when outside a Git tree). That rule keeps nested Compose layouts such as deploy/edge/docker-compose.yml attributed to their enclosing stack repository rather than to edge. When multiple stacks use the same image (for example postgres:18.7-alpine), byRef deduplicates the scan target so Trivy scans the image once while recording every consuming stack in Image.Stacks.
Trivy execution, offline ingestion, and partial-failure handling (internal/trivy)
Section titled “Trivy execution, offline ingestion, and partial-failure handling (internal/trivy)”vigil defines a two-method Scanner interface with two implementations:
ExecScanner(live mode): invokestrivy image --scanners vuln --quiet --skip-version-check --format json <image>(plus--severitywhen configured) and unmarshalsArtifactNameandResults[].Vulnerabilities[]1.DirScanner(offline and test mode): activated when--results-diris passed tovigil scan(used bymake sweepagainst/tmp/trivy-sweep). It reads<dir>/<sanitized>.json, whereSanitizeNamereplaces/,:, and@with_(matchingtr '/:@' '___').
If scanner.Scan returns an error for an individual image (for example, a private image when registry credentials are absent or a tag removed upstream), cmdScan records the error string on ImageResult.Error, logs a warning, and continues scanning the remaining images. In the rendered Markdown report, failed scans appear in the summary header (N images, M scan failures) and render a scan failed: <err> row in the images table rather than aborting the sweep or masquerading as zero-vulnerability images.
Exploitability enrichment (internal/enrich)
Section titled “Exploitability enrichment (internal/enrich)”Once all images are scanned, cmdScan collects every unique vulnerability ID with a CVE- prefix (skipping non-CVE advisory identifiers such as GHSA-* or DLA-*) and calls enrichState unless --offline is set:
- CISA KEV catalog:
enrich.LoadKEVperforms an HTTPGETtohttps://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json2 and builds amap[string]boolfromvulnerabilities[].cveID. - FIRST EPSS v1 API:
enrich.LoadEPSSslices the deduplicated CVE list into chunks of100(epssChunk = 100) and querieshttps://api.first.org/data/v1/epss?cve=CVE-...,CVE-...3, parsing each returneddata[].epssstring into afloat64in[0.0, 1.0].
Both HTTP calls use a 30s client timeout and wrap the response body in io.LimitReader(resp.Body, 64<<20) (64 MiB). If either endpoint times out or returns a non-200 status, enrichState logs slog.Warn and substitutes an empty map so the sweep still writes .vigil/state.json with Trivy severities and exposure tiers intact.
Exposure tiers and the unknown-first invariant (internal/tier)
Section titled “Exposure tiers and the unknown-first invariant (internal/tier)”A vulnerability’s severity in a vacuum does not tell the operator whether an attacker can reach the affected port. stacks.yaml maps each stack directory name to one of three explicit tiers, while internal/tier adds a fourth unknown tier for any stack not listed in the file:
| Tier | Rank() | Meaning | Examples from stacks.yaml |
|---|---|---|---|
unknown | 0 (sorts first) | Stack is absent from stacks.yaml, or the image has no consuming stack (an --images-file reference that appears in no discovered Compose file); untriaged exposure | Any newly added stack before stacks.yaml is updated |
public | 1 | Reachable from the internet through the edge Caddy reverse proxy | The edge proxy stack, a photo-library stack, a web-research stack, the media-server stack, and the password-manager, forge, and SSO stacks |
lan | 2 | Reachable on the LAN or tailnet only, with no public virtual host | The session-history store, the transcription service, a DNS resolver, a local-LLM research assistant, and the monitoring and object-storage stacks |
internal | 3 | Bridge-network only or test fixture with no published path | A tailnet-only proxy, a WAF test fixture |
When an image is used by multiple stacks, Registry.ForStacks(stacks) computes the effective tier across all consumers:
func (r *Registry) ForStacks(stacks []string) Tier { best := TierInternal if len(stacks) == 0 { return TierUnknown } for _, s := range stacks { if t := r.Lookup(s); t == TierUnknown { return TierUnknown // any unclassified consumer poisons the join } else if t.Rank() < best.Rank() && t != TierUnknown { best = t } } return best}Two properties of ForStacks are deliberate:
- Most-exposed consumer wins: if
valkey/valkeyruns in both alanstack and apublicstack, the image is rankedpublic. - Any
unknownconsumer poisons the join: if an image is shared between aninternalstack and a newly created stack that has not been added tostacks.yaml,ForStacksreturnsunknown(rank0) so the image ranks ahead of every tiered image with the same KEV count until the new stack is classified. Likewise, ifstacks.yamlis missing on disk,tier.Loadreturns an empty registry without error, causing every image to report asunknown.
State snapshots, sweep deltas, and report ordering (internal/store, internal/report)
Section titled “State snapshots, sweep deltas, and report ordering (internal/store, internal/report)”store.Save marshals store.State (generated_at timestamp plus images[]) as indented JSON to <path>.tmp and renames it over <path> (default .vigil/state.json). No database daemon is used: state lives as a flat JSON file alongside the weekly job that produces it.
When vigil report is invoked with --prev <prev.json>, store.Diff(prev, curr) flattens all findings in each snapshot into a map keyed by image|id|pkg:
- New findings (
Delta.New):(image, ID, Pkg)keys present incurrbut absent fromprev, sorted lexicographically by key. - Resolved findings (
Delta.Resolved):(image, ID, Pkg)keys present inprevbut absent fromcurr, sorted lexicographically by key.
report.Markdown(state, delta) renders three sections in priority order:
## Known-exploited (CISA KEV): N findings: every finding withKEV == trueacross the fleet, sorted bytrivy.SeverityRank(CRITICAL->HIGH->MEDIUM->LOW) and then by descendingEPSSscore, showingimage | tier | cve | package | epss | fix available.## Delta: N new, M resolved(when--previs supplied): summary counts followed by the### New findingstable (image | cve | package).## Images (worst first): every scanned image (image | tier | CRIT | HIGH | MED | KEV | max EPSS), ordered byreport.SortImages:- Higher KEV finding count first (
kevCount(a) > kevCount(b)). - Lower exposure
Tier.Rank()first (unknown->public->lan->internal). - Higher
CRITICALcount first, then higherHIGHcount first.
- Higher KEV finding count first (
CI gating and how vigil pairs with Renovate
Section titled “CI gating and how vigil pairs with Renovate”While vigil scan and vigil report run across the whole fleet (including unfixed CVEs so KEV hits without upstream patches remain visible), vigil gate is built for per-repository CI pipelines:
vigil gate --image ghcr.io/example/vigil:dev --severity CRITICAL,HIGH --ignore-unfixedcmdGate scans a single image via ExecScanner, filters rep.Vulns() to severities matching --severity (default CRITICAL,HIGH), discards findings with an empty FixedVersion when --ignore-unfixed is true (the default), sorts any remaining findings by SeverityRank, prints each offending Severity | ID | PkgName InstalledVersion -> FixedVersion line, and exits 1.
- At first-party image build time (CI gate): repositories that build and push container images (
vigil, a tailnet-only proxy, a release-tagged service, a web-research stack) run Trivy directly with--severity CRITICAL,HIGH --ignore-unfixed --exit-code 1against a locally built candidate image; the job fails before the registry push when it finds a fixable CRITICAL or HIGH. The multi-architecture images pushed afterwards are not scanned again. - Daily at
18:40UTC (self-hosted Renovate): Renovate checks upstream registries andgo.modfiles, opening pull requests or Dependency Dashboard checkboxes when new versions exist. - Weekly across the fleet (
vigil scan+vigil report --prev):vigilscans all declared Compose images (including third-party images that do not run a CI build in Forgejo) and ranks findings by KEV, exposure tier, severity, and EPSS so the operator knows which image updates or base-image rebuilds remove real risk.
vigil’s own Forgejo Actions workflows
Section titled “vigil’s own Forgejo Actions workflows”vigil applies the same checks to its own repository on the Forgejo Actions runner:
.forgejo/workflows/ci.yml: runs onpushtomain,pull_request, andworkflow_dispatch. It turns offsetup-go’s built-in caching (cache: false, because Go build and module caches on the self-hosted runner are already persistent shared volumes), runsgo vet ./..., checksgofmt -l ., runsgo test ./... -race -count=1, and executesgovulncheck ./...4 for call-graph reachability analysis against the Go vulnerability database..forgejo/workflows/docker.yml: builds a local candidate image (vigil:ci-<sha>,load: true) on the sharedci-builderBuildKit instance (network=forgejo_forgejo,provenance-add-gha=falseso the persistent builder does not bake the creating job’s event payload into subsequent builds), downloads the standalonetrivy0.74.0release tarball to/tmp/trivy.tgz, extracts the binary as/tmp/trivy, and gates the local image before building and pushing the multi-architecture (linux/amd64,linux/arm64)gcr.io/distroless/static-debian12:nonrootruntime image toghcr.io.renovate.jsonandDockerfileconventions:vigil/renovate.jsonopts into"enabledManagers": ["dockerfile", "gomod"]with"postUpdateOptions": ["gomodTidy"]and groups Go minor and patch updates into a single"go modules"pull request. ItsDockerfilepinsFROM golang:1.27-alpine AS builderdirectly on theFROMline without anARGindirection, because Renovate substitutes anARGdefault when evaluatingFROMand then fails to locate the literal string to rewrite.
Design invariants and planned extensions
Section titled “Design invariants and planned extensions”The repository’s AGENTS.md enforces six design invariants:
- Never match CVEs in
vigilitself: advisory databases and version comparisons belong in Trivy;vigilonly orchestrates, enriches, joins, and diffs. - Stdlib-first: every dependency in
go.modis another package thatgovulncheckand Trivy must scan.gopkg.in/yaml.v3is the sole external dependency. unknowntier sorts first: unclassified exposure (Rank() == 0) always outrankspublic(Rank() == 1).- Feed failures degrade rather than fail: an outage at
cisa.govorapi.first.orgloses only that feed’s enrichment and logs a warning, never a failed sweep. --ignore-unfixedis the default gate posture: CI gates block on vulnerabilities that have a published fix (FixedVersion != "").- State is a file:
.vigil/state.jsonand set-difference deltas avoid running a database service for a weekly batch job.
Planned extensions (not yet implemented)
Section titled “Planned extensions (not yet implemented)”Three precision and packaging improvements are recorded in TODO.md and docs/plans/2026-08-20-security-pipeline-completion.md as planned work:
- Docker-socket inventory (
--docker-socket, planned):inventory.Collectcurrently walks Compose files on disk, which reflects what repositories declare rather than containers started manually or running stale tags. The plannedCollectDockersource unionsdocker ps --format '{{.Image}}'intobyRefwith a synthetic<host>:runningstack entry so undeclared running containers report with tierunknownat the top of the digest. - Per-image tier overrides in
stacks.yaml(planned): becausestacks.yamlcurrently assigns tiers at the stack directory level, internal helper containers inside a public stack (such as a CF-challenge solver helper and a music-library download helper inside a media-server stack) conservatively inherit tierpublic. Adding an optional top-levelimages:map tostacks.yamlwill let per-image overrides take precedence over the stack join. - Self-contained sweeper container and extra flags (planned):
vigil’s runtimeDockerfile(gcr.io/distroless/static-debian12:nonroot) ships CA certificates for the KEV and EPSS HTTPS feeds and/usr/local/bin/vigil, but does not bundle atrivybinary (scanmode either runs on a host withtrivyinstalled or ingests--results-dir). A multi-stage sweeper image variant, an EPSS threshold flag, and floating-tag detection (postgres:17) are tracked on the board.
Related docs
Section titled “Related docs”- Self-hosted Renovate for a Forgejo-primary Docker Compose fleet - the daily update proposer that opens pull requests for image and Go module bumps
- Porting GitHub Actions to Forgejo Actions - runner constraints, shared BuildKit caching (
ci-builder), and workflow patterns used byvigil’s CI - Forgejo-primary fleet migration - moving the fleet to Forgejo as primary and replacing GitHub Dependabot
- Composer GitOps Docker Compose manager - the stack manager and pipeline scheduler that runs the weekly router sweep
- Self-hosted Forgejo on the router - the Forgejo instance and Actions runner that execute
vigil’s CI and image gate
References
Section titled “References”-
Aqua Security, “Vulnerability Scanning,” Trivy Documentation. https://trivy.dev/docs/latest/scanner/vulnerability/ ↩ ↩2 ↩3
-
Cybersecurity and Infrastructure Security Agency, “Known Exploited Vulnerabilities Catalog,” CISA. https://www.cisa.gov/known-exploited-vulnerabilities-catalog ↩ ↩2 ↩3 ↩4
-
Forum of Incident Response and Security Teams, “Exploit Prediction Scoring System (EPSS),” FIRST. https://www.first.org/epss/ ↩ ↩2 ↩3 ↩4
-
Go Authors, “govulncheck command - golang.org/x/vuln/cmd/govulncheck,” Go Packages. https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck ↩