Skip to content

Forge and CIreference

Vigil fleet vulnerability sweep, exploitability enrichment, and CI gating

Scanning a homelab container fleet with raw vulnerability scanners produces long lists of unranked CVE rows, most of them unfixable Debian or Alpine base-layer advisories on services that are not reachable from the internet. vigil is a single-binary Go CLI (Go, with gopkg.in/yaml.v3 as its sole external dependency) that orchestrates Trivy1 across Docker Compose trees, enriches every CVE with CISA Known Exploited Vulnerabilities (KEV)2 membership and FIRST Exploit Prediction Scoring System (EPSS)3 probabilities, joins each image onto a static exposure registry (stacks.yaml), and diffs consecutive sweeps so the weekly digest highlights what changed and which findings are both reachable and exploitable.

The vigil CLI and weekly router sweep pipeline launched on 2026-08-20; its CI and multi-architecture container build workflows moved from GitHub Actions to the self-hosted Forgejo instance on 2026-11-16, where vigil also opted into the self-hosted Renovate sweep for Dockerfile and Go module updates.

  • Trivy for CVE matching, vigil for enrichment and ranking: vigil delegates package and OS advisory matching to trivy image --scanners vuln1 (ExecScanner or offline DirScanner), adding exploitability context, reachability ranking, and state diffs on top of Trivy’s JSON output.
  • Runtime exploitability feeds: vigil scan fetches the CISA KEV catalog2 and queries the FIRST EPSS v1 API3 in batches of 100 CVEs at run time. Each feed degrades on its own: if KEV is unreachable, KEV flags stay false; if EPSS is unreachable, scores stay zero. The sweep logs a warning and still writes the state file.
  • unknown exposure sorts first: stacks.yaml classifies stacks as public (rank 1), lan (rank 2), or internal (rank 3). Any stack missing from stacks.yaml receives tier unknown (rank 0). The report orders images by KEV hit count first, then tier rank, so among images with equal KEV counts the unclassified ones sort above public and force triage instead of sinking to the bottom.
  • --ignore-unfixed by default in CI gates: vigil gate fails a build only on CRITICAL or HIGH vulnerabilities that carry a non-empty FixedVersion, preventing unfixable base-image advisories from training operators to bypass the gate.
  • File state and set-difference deltas: each sweep writes .vigil/state.json atomically via a .tmp rename, and vigil report --prev computes the set difference on (image, vulnerability ID, package) tuples so weekly reviews start from newly appeared and newly resolved findings.
  • Paired with Renovate: self-hosted Renovate opens pull requests when upstream tags move; vigil shows which images carry actively exploited or high-EPSS vulnerabilities, how exposed they are and which fixed versions exist, so the operator knows which image update or base-image rebuild to take first. It does not read Renovate’s pull requests itself.
Compose trees+ --images-file(internal/inventory)Trivy CLI or--results-dir JSON(internal/trivy)deduped refs(skip :local).vigil/state.json+ prev.json diff(internal/store)per-imageVulns + CountsCISA KEV JSON +FIRST EPSS API(internal/enrich)KEV bool +EPSS float64Exposure registrystacks.yaml(internal/tier)unknown | public |lan | internalMarkdown digestor JSON output(internal/report)KEV -> tier ->CRIT -> HIGH

The codebase is split into a thin CLI entry point (cmd/vigil/main.go) and six internal packages with no circular dependencies:

Package / pathResponsibilityKey types and entry points
cmd/vigil/main.goSubcommand dispatch (scan, report, gate, healthcheck) and signal handling (SIGINT, SIGTERM)cmdScan, cmdReport, cmdGate
internal/inventoryWalks a directory tree for Compose files, resolves ${VAR:-default} expressions, merges --images-file, and attributes each image to its consuming stacksImage, Collect(root, imagesFile)
internal/trivyRuns trivy image (ExecScanner) or reads cached JSON files (DirScanner), parses Results[].Vulnerabilities, and ranks severitiesReport, Vuln, Scanner, ExecScanner, DirScanner, SanitizeName
internal/enrichFetches the CISA KEV catalog and batch-queries the FIRST EPSS API (100 CVEs per request, 30s HTTP timeout, 64 MiB body cap)LoadKEV, LoadEPSS, KEVURL, EPSSURL
internal/tierParses stacks.yaml and resolves the effective exposure tier across all stacks consuming an imageTier, Registry, Load, ForStacks
internal/storeSerialises sweep state atomically (.tmp write + os.Rename) and computes New and Resolved deltas between two snapshotsState, ImageResult, Vuln, FindingRef, Delta, Diff
internal/reportSorts images and KEV findings and renders the Markdown digest or formatted JSONSortImages, Markdown(state, delta), JSON(state)

Why orchestrate Trivy instead of reading raw scanner output

Section titled “Why orchestrate Trivy instead of reading raw scanner output”
Question during triage or CIRaw trivy image outputvigil (scan, report, gate)
Which reported CVEs are actively exploited in the wild?Not distinguished from theoretical CVEs with the same CVSS severityAnnotates every CVE-* with CISA KEV membership2 and FIRST EPSS 30-day exploitation probability3, rendering KEV hits in a dedicated top table
Is the affected container reachable from the internet or isolated on an internal bridge?Scanner only sees the image reference, not how stacks deploy itJoins every image against stacks.yaml (unknown, public, lan, internal) and promotes shared images to their most-exposed consumer’s tier
What changed since last week’s sweep?Requires diffing per-image outputs by handstore.Diff diffs .vigil/state.json against --prev on image|id|pkg keys; the report lists new findings and counts resolved ones
Should this CVE fail a CI image build?Failing on all CRITICAL,HIGH blocks deploys on unpatched Debian/Alpine base layersvigil gate defaults to --severity CRITICAL,HIGH --ignore-unfixed=true, failing only when FixedVersion != ""
What happens when a new Compose stack is added to the host?Unnoticed unless added to a scan listDirectory walk discovers it automatically; if omitted from stacks.yaml, it receives tier unknown and sorts first in the report

Image discovery and stack attribution (internal/inventory)

Section titled “Image discovery and stack attribution (internal/inventory)”

inventory.Collect(root, imagesFile) builds the deduplicated list of container images to scan from two sources: a recursive walk of --root and an optional newline-separated --images-file.

During the directory walk, Collect skips .git, node_modules, .vigil, backup, and backups directories and parses any file matching ^(docker-)?compose(-[\w.]+)?\.ya?ml$. For each service’s image: string, resolve() performs three normalisation steps:

  1. Strips inline comments beginning with " #".
  2. Expands ${VAR:-default} variable expressions to their inline default string using the regex \$\{([A-Za-z_][A-Za-z0-9_]*)(?::-([^}]*))?\}.
  3. Rejects any reference that still contains an unresolved ${VAR} (no default value) or ends with :local (host-built images that do not exist in a registry).

To attribute a Compose file to a stack name, stackName(path, root) walks upward from the Compose file’s directory toward root until it finds a directory containing a .git/ subdirectory, returning that repository root’s basename (or the immediate parent directory’s basename when outside a Git tree). That rule keeps nested Compose layouts such as deploy/edge/docker-compose.yml attributed to their enclosing stack repository rather than to edge. When multiple stacks use the same image (for example postgres:18.7-alpine), byRef deduplicates the scan target so Trivy scans the image once while recording every consuming stack in Image.Stacks.

Trivy execution, offline ingestion, and partial-failure handling (internal/trivy)

Section titled “Trivy execution, offline ingestion, and partial-failure handling (internal/trivy)”

vigil defines a two-method Scanner interface with two implementations:

  • ExecScanner (live mode): invokes trivy image --scanners vuln --quiet --skip-version-check --format json <image> (plus --severity when configured) and unmarshals ArtifactName and Results[].Vulnerabilities[]1.
  • DirScanner (offline and test mode): activated when --results-dir is passed to vigil scan (used by make sweep against /tmp/trivy-sweep). It reads <dir>/<sanitized>.json, where SanitizeName replaces /, :, and @ with _ (matching tr '/:@' '___').

If scanner.Scan returns an error for an individual image (for example, a private image when registry credentials are absent or a tag removed upstream), cmdScan records the error string on ImageResult.Error, logs a warning, and continues scanning the remaining images. In the rendered Markdown report, failed scans appear in the summary header (N images, M scan failures) and render a scan failed: <err> row in the images table rather than aborting the sweep or masquerading as zero-vulnerability images.

Exploitability enrichment (internal/enrich)

Section titled “Exploitability enrichment (internal/enrich)”

Once all images are scanned, cmdScan collects every unique vulnerability ID with a CVE- prefix (skipping non-CVE advisory identifiers such as GHSA-* or DLA-*) and calls enrichState unless --offline is set:

  1. CISA KEV catalog: enrich.LoadKEV performs an HTTP GET to https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json2 and builds a map[string]bool from vulnerabilities[].cveID.
  2. FIRST EPSS v1 API: enrich.LoadEPSS slices the deduplicated CVE list into chunks of 100 (epssChunk = 100) and queries https://api.first.org/data/v1/epss?cve=CVE-...,CVE-...3, parsing each returned data[].epss string into a float64 in [0.0, 1.0].

Both HTTP calls use a 30s client timeout and wrap the response body in io.LimitReader(resp.Body, 64<<20) (64 MiB). If either endpoint times out or returns a non-200 status, enrichState logs slog.Warn and substitutes an empty map so the sweep still writes .vigil/state.json with Trivy severities and exposure tiers intact.

Exposure tiers and the unknown-first invariant (internal/tier)

Section titled “Exposure tiers and the unknown-first invariant (internal/tier)”

A vulnerability’s severity in a vacuum does not tell the operator whether an attacker can reach the affected port. stacks.yaml maps each stack directory name to one of three explicit tiers, while internal/tier adds a fourth unknown tier for any stack not listed in the file:

TierRank()MeaningExamples from stacks.yaml
unknown0 (sorts first)Stack is absent from stacks.yaml, or the image has no consuming stack (an --images-file reference that appears in no discovered Compose file); untriaged exposureAny newly added stack before stacks.yaml is updated
public1Reachable from the internet through the edge Caddy reverse proxyThe edge proxy stack, a photo-library stack, a web-research stack, the media-server stack, and the password-manager, forge, and SSO stacks
lan2Reachable on the LAN or tailnet only, with no public virtual hostThe session-history store, the transcription service, a DNS resolver, a local-LLM research assistant, and the monitoring and object-storage stacks
internal3Bridge-network only or test fixture with no published pathA tailnet-only proxy, a WAF test fixture

When an image is used by multiple stacks, Registry.ForStacks(stacks) computes the effective tier across all consumers:

func (r *Registry) ForStacks(stacks []string) Tier {
best := TierInternal
if len(stacks) == 0 {
return TierUnknown
}
for _, s := range stacks {
if t := r.Lookup(s); t == TierUnknown {
return TierUnknown // any unclassified consumer poisons the join
} else if t.Rank() < best.Rank() && t != TierUnknown {
best = t
}
}
return best
}

Two properties of ForStacks are deliberate:

  • Most-exposed consumer wins: if valkey/valkey runs in both a lan stack and a public stack, the image is ranked public.
  • Any unknown consumer poisons the join: if an image is shared between an internal stack and a newly created stack that has not been added to stacks.yaml, ForStacks returns unknown (rank 0) so the image ranks ahead of every tiered image with the same KEV count until the new stack is classified. Likewise, if stacks.yaml is missing on disk, tier.Load returns an empty registry without error, causing every image to report as unknown.

State snapshots, sweep deltas, and report ordering (internal/store, internal/report)

Section titled “State snapshots, sweep deltas, and report ordering (internal/store, internal/report)”

store.Save marshals store.State (generated_at timestamp plus images[]) as indented JSON to <path>.tmp and renames it over <path> (default .vigil/state.json). No database daemon is used: state lives as a flat JSON file alongside the weekly job that produces it.

When vigil report is invoked with --prev <prev.json>, store.Diff(prev, curr) flattens all findings in each snapshot into a map keyed by image|id|pkg:

  • New findings (Delta.New): (image, ID, Pkg) keys present in curr but absent from prev, sorted lexicographically by key.
  • Resolved findings (Delta.Resolved): (image, ID, Pkg) keys present in prev but absent from curr, sorted lexicographically by key.

report.Markdown(state, delta) renders three sections in priority order:

  1. ## Known-exploited (CISA KEV): N findings: every finding with KEV == true across the fleet, sorted by trivy.SeverityRank (CRITICAL -> HIGH -> MEDIUM -> LOW) and then by descending EPSS score, showing image | tier | cve | package | epss | fix available.
  2. ## Delta: N new, M resolved (when --prev is supplied): summary counts followed by the ### New findings table (image | cve | package).
  3. ## Images (worst first): every scanned image (image | tier | CRIT | HIGH | MED | KEV | max EPSS), ordered by report.SortImages:
    • Higher KEV finding count first (kevCount(a) > kevCount(b)).
    • Lower exposure Tier.Rank() first (unknown -> public -> lan -> internal).
    • Higher CRITICAL count first, then higher HIGH count first.

CI gating and how vigil pairs with Renovate

Section titled “CI gating and how vigil pairs with Renovate”

While vigil scan and vigil report run across the whole fleet (including unfixed CVEs so KEV hits without upstream patches remain visible), vigil gate is built for per-repository CI pipelines:

Terminal window
vigil gate --image ghcr.io/example/vigil:dev --severity CRITICAL,HIGH --ignore-unfixed

cmdGate scans a single image via ExecScanner, filters rep.Vulns() to severities matching --severity (default CRITICAL,HIGH), discards findings with an empty FixedVersion when --ignore-unfixed is true (the default), sorts any remaining findings by SeverityRank, prints each offending Severity | ID | PkgName InstalledVersion -> FixedVersion line, and exits 1.

Where in the lifecycle?1. CI image build on push(trivy image --ignore-unfixed --exit-code 1)-> fail before the registry pushon fixable CRIT/HIGHfirst-party image build2. Daily 18:40 UTC registry check(self-hosted Renovate)-> open PRs for new image tags + gomodupstream release published3. Weekly fleet sweep(vigil scan + vigil report --prev)-> rank KEV, EPSS, exposure tier, and deltaadvisory published between deploysOperator merges Renovate PRor bumps base image / Dockerfileprioritises which PRsand Dockerfiles matter
  1. At first-party image build time (CI gate): repositories that build and push container images (vigil, a tailnet-only proxy, a release-tagged service, a web-research stack) run Trivy directly with --severity CRITICAL,HIGH --ignore-unfixed --exit-code 1 against a locally built candidate image; the job fails before the registry push when it finds a fixable CRITICAL or HIGH. The multi-architecture images pushed afterwards are not scanned again.
  2. Daily at 18:40 UTC (self-hosted Renovate): Renovate checks upstream registries and go.mod files, opening pull requests or Dependency Dashboard checkboxes when new versions exist.
  3. Weekly across the fleet (vigil scan + vigil report --prev): vigil scans all declared Compose images (including third-party images that do not run a CI build in Forgejo) and ranks findings by KEV, exposure tier, severity, and EPSS so the operator knows which image updates or base-image rebuilds remove real risk.

vigil applies the same checks to its own repository on the Forgejo Actions runner:

  • .forgejo/workflows/ci.yml: runs on push to main, pull_request, and workflow_dispatch. It turns off setup-go’s built-in caching (cache: false, because Go build and module caches on the self-hosted runner are already persistent shared volumes), runs go vet ./..., checks gofmt -l ., runs go test ./... -race -count=1, and executes govulncheck ./...4 for call-graph reachability analysis against the Go vulnerability database.
  • .forgejo/workflows/docker.yml: builds a local candidate image (vigil:ci-<sha>, load: true) on the shared ci-builder BuildKit instance (network=forgejo_forgejo, provenance-add-gha=false so the persistent builder does not bake the creating job’s event payload into subsequent builds), downloads the standalone trivy 0.74.0 release tarball to /tmp/trivy.tgz, extracts the binary as /tmp/trivy, and gates the local image before building and pushing the multi-architecture (linux/amd64,linux/arm64) gcr.io/distroless/static-debian12:nonroot runtime image to ghcr.io.
  • renovate.json and Dockerfile conventions: vigil/renovate.json opts into "enabledManagers": ["dockerfile", "gomod"] with "postUpdateOptions": ["gomodTidy"] and groups Go minor and patch updates into a single "go modules" pull request. Its Dockerfile pins FROM golang:1.27-alpine AS builder directly on the FROM line without an ARG indirection, because Renovate substitutes an ARG default when evaluating FROM and then fails to locate the literal string to rewrite.

The repository’s AGENTS.md enforces six design invariants:

  1. Never match CVEs in vigil itself: advisory databases and version comparisons belong in Trivy; vigil only orchestrates, enriches, joins, and diffs.
  2. Stdlib-first: every dependency in go.mod is another package that govulncheck and Trivy must scan. gopkg.in/yaml.v3 is the sole external dependency.
  3. unknown tier sorts first: unclassified exposure (Rank() == 0) always outranks public (Rank() == 1).
  4. Feed failures degrade rather than fail: an outage at cisa.gov or api.first.org loses only that feed’s enrichment and logs a warning, never a failed sweep.
  5. --ignore-unfixed is the default gate posture: CI gates block on vulnerabilities that have a published fix (FixedVersion != "").
  6. State is a file: .vigil/state.json and set-difference deltas avoid running a database service for a weekly batch job.

Three precision and packaging improvements are recorded in TODO.md and docs/plans/2026-08-20-security-pipeline-completion.md as planned work:

  • Docker-socket inventory (--docker-socket, planned): inventory.Collect currently walks Compose files on disk, which reflects what repositories declare rather than containers started manually or running stale tags. The planned CollectDocker source unions docker ps --format '{{.Image}}' into byRef with a synthetic <host>:running stack entry so undeclared running containers report with tier unknown at the top of the digest.
  • Per-image tier overrides in stacks.yaml (planned): because stacks.yaml currently assigns tiers at the stack directory level, internal helper containers inside a public stack (such as a CF-challenge solver helper and a music-library download helper inside a media-server stack) conservatively inherit tier public. Adding an optional top-level images: map to stacks.yaml will let per-image overrides take precedence over the stack join.
  • Self-contained sweeper container and extra flags (planned): vigil’s runtime Dockerfile (gcr.io/distroless/static-debian12:nonroot) ships CA certificates for the KEV and EPSS HTTPS feeds and /usr/local/bin/vigil, but does not bundle a trivy binary (scan mode either runs on a host with trivy installed or ingests --results-dir). A multi-stage sweeper image variant, an EPSS threshold flag, and floating-tag detection (postgres:17) are tracked on the board.
  1. Aqua Security, “Vulnerability Scanning,” Trivy Documentation. https://trivy.dev/docs/latest/scanner/vulnerability/ ↩ ↩2 ↩3

  2. Cybersecurity and Infrastructure Security Agency, “Known Exploited Vulnerabilities Catalog,” CISA. https://www.cisa.gov/known-exploited-vulnerabilities-catalog ↩ ↩2 ↩3 ↩4

  3. Forum of Incident Response and Security Teams, “Exploit Prediction Scoring System (EPSS),” FIRST. https://www.first.org/epss/ ↩ ↩2 ↩3 ↩4

  4. Go Authors, “govulncheck command - golang.org/x/vuln/cmd/govulncheck,” Go Packages. https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck ↩