Rotating a credential end to end with secretctl

Rotating a credential by hand is a chain of five separate mistakes waiting to happen: the new value gets echoed while you are generating it, set writes it to the wrong file, the commit ships plaintext, the deploy hands the running container ciphertext, or nobody checks that every host actually picked up the new value. This guide walks the chain once, correctly, with the commands and their exact output at each step. It assumes the registry-and-guard design described in Keeping credentials out of coding-agent transcripts; Part 1 and Part 5 below are the how-to companion to that reference doc’s architecture.

Prerequisites: secretctl on PATH, a destination you can write to (a dotenv file, a sops-encrypted dotenv, a key file, or a Bitwarden item via bw serve), and, if the rotation deploys, a Composer-managed stack. Every command below ran against a canary value - a throwaway credential generated for this guide and never a real one - in an isolated scratch directory with its own registry, so none of it touched a real store.

Component	Version at time of writing
secretctl	built 2026-09-29; set carries the 2026-09-29 dotenv-canonicalisation fix verified in Part 2

Architecture

Text fallback for the diagram:

1. Mint the new value and capture it by reference (a file or a shell variable), never by printing it.
2. secretctl set writes it into a destination and prints a before and after digest.
3. The destination is a dotenv, key file, sops-encrypted dotenv, or Bitwarden item.
4. If the destination is a committed file, the global pre-commit hook refuses it unless it is already encrypted.
5. A Composer deploy decrypts the stack’s .env, brings the containers up, and re-encrypts it.
6. fp --salt-file and cmp prove the new value reached every host that is supposed to hold it.
7. The destination is also what the secret-output guard’s digest cache reads, on its own schedule (Part 5).

Part 1: mint the value without it touching the transcript

The registry-based guard described in the reference doc above only masks a value it already knows: a fresh HMAC digest exists for every store in the registry, but a credential minted during the session has no store yet, so nothing masks it. On 2026-09-20 a session generated a Forgejo access token and echoed it into its own transcript for exactly this reason - the guard had nothing to compare it against.

The fix is capture-by-reference: write the freshly generated value straight to a file or a shell variable, use it only by reference from then on, and register its store before it is used a second time.

TOK=$(some-generator --raw)
printf '%s' "$TOK" > canary.key
chmod 600 canary.key
curl -H "Authorization: token $TOK" https://example/api

Never echo "$TOK" and never paste it into a command as a literal. Once the store is registered (a line in ~/.config/secretctl/sources), the guard can mask it in every later tool call; until then it is invisible by construction, not because a rule failed to fire.

The worked example in this guide follows the same discipline: every value is a canary generated with openssl rand -hex 24 inside a scratch directory, captured straight to a file, and never printed. Only digests, exit codes, and refusal messages appear below.

Part 2: secretctl set and its per-destination rules

secretctl set <dst> --from <src> moves a value into a destination without ever putting it on stdout. It always prints a before-digest and an after-digest to stderr, plus the canonicalisation rule in force, so a rotation that silently did nothing is visible as an unchanged digest rather than a quiet success:

$ secretctl set dotenv:.../new.env#CANARY_TOKEN --from keyfile:.../canary.key
canonical: strip at most one trailing \n (and a preceding \r); bytes otherwise verbatim

  dotenv:.../new.env#CANARY_TOKEN
    before  74f1963a4085 (48B)
    after   2ec1f2cce7d0 (48B)

exit: 0

Setting a value to its current value is legitimate, but set says so rather than leaving before and after identical with no comment: note: the digest did not change. Either the value was already this, or the write did not land.

--from refuses a remote source outright (docker:HOST/example#VAR, sshenv:..., uci:...): writing from one would require pulling the plaintext onto the local host first, which is exactly what the remote path exists to avoid. The refusal fires before any network round trip:

cannot set from a remote source (docker:HOST/example#VAR): that would transport the plaintext

What each destination scheme does

Destination	What set does	Refusal
dotenv:PATH#KEY	Upserts the KEY=VALUE line in place, preserving comments, blank lines, and any export prefix	a multi-line value (see below)
sops:PATH#KEY	Decrypts in memory, upserts the field, re-encrypts to the file’s OWN age recipients (never a .sops.yaml creation rule); dotenv-format sops files only	a .yaml/.yml/.json destination: sops set supports dotenv-format files only; PATH looks like structured config
keyfile:PATH	Overwrites the whole file atomically with the source’s bytes, verbatim - it never runs the dotenv codec	none
bw:ITEM#FIELD	Updates an existing non-empty custom field in place, or upserts a FIELD=value line in the notes body	a field-less bw:ITEM destination, refused outright: rewriting the whole notes body would destroy every other credential the item holds
rbw:ITEM[#FIELD]	Never - rbw edits go through an interactive $EDITOR, which set cannot drive	rbw destination is read-only (rbw edits via an interactive $EDITOR); write the vault with bw_set (bw serve) or the web vault, then read it back as rbw:ITEM
env:NAME	Never - a child process cannot write into its parent’s environment, so there is no way for set to make a value appear in the calling shell	destination scheme "env" is not writable (writable: dotenv, keyfile, sops, bw)

secretctl explain <src> shows how a source would resolve, including the remote shell command it would run, without touching or printing the value - a way to audit a remote query before running it.

Because a caller must never parse the printed table, secretctl uses one exit-code contract throughout fp, cmp, and set: 0 is MATCH (or a clean write), 1 is MISMATCH, 2 is UNRESOLVED or a refusal. An UNRESOLVED comparison - a typo’d store name - must never read as a mismatch, and an empty-vs-empty resolve is deliberately UNRESOLVED rather than MATCH, because two empty values would otherwise digest identically, the most dangerous false MATCH available.

The dotenv newline rule, and its 2026-09-29 fix

A dotenv line cannot represent a newline: writing one anyway produces a file that parses back to a truncated credential. set refuses a multi-line value into a dotenv: or sops: dotenv destination.

Until 2026-09-29 that check ran on the raw source bytes, with no canonicalisation - so a key file produced the ordinary way, openssl rand -hex 24 > file (which appends one trailing newline), refused to set into a dotenv destination at all, even though fp and cmp on the same file reported a clean MATCH against a dotenv holding “the same” value, because their comparison strips at most one trailing newline before hashing. As of 2026-09-29, set applies that same canonicalisation - credential.Canonical, the rule fp/cmp already use - before the newline check, so the two commands agree on what counts as the same value. Re-run this run against the fixed binary (HEAD 730896f4), a fresh canary key file with one ordinary trailing newline now sets cleanly:

$ secretctl set dotenv:.../new.env#CANARY_TOKEN --from keyfile:.../canary.key
  dotenv:.../new.env#CANARY_TOKEN
    before  74f1963a4085 (48B)
    after   2ec1f2cce7d0 (48B)
exit: 0

$ secretctl cmp dotenv:.../new.env#CANARY_TOKEN keyfile:.../canary.key
  dotenv:.../new.env#CANARY_TOKEN  MATCH  a480da077fb9 (48B)
  keyfile:.../canary.key           MATCH  a480da077fb9 (49B)
=== MATCH: every source holds the same credential. ===
note: byte lengths differ - same credential, different framing (a trailing newline or quoting). Harmless here; will confuse any reader that does not canonicalise.
exit: 0

The fix strips at most one trailing newline (and a preceding carriage return); it does not open the door to anything wider. A value with an interior newline is still refused, and so is a value with two trailing newlines - the canonicalisation removes only the first:

$ secretctl set dotenv:.../new.env#CANARY_TOKEN --from keyfile:.../interior.val
secretctl set: value for CANARY_TOKEN contains a newline; dotenv cannot represent it
exit: 2

$ secretctl set dotenv:.../new.env#CANARY_TOKEN --from keyfile:.../double.val
secretctl set: value for CANARY_TOKEN contains a newline; dotenv cannot represent it
exit: 2

keyfile: DESTINATIONS were never affected either way: they bypass the dotenv codec entirely and always write the source’s bytes verbatim, so a keyfile-to-keyfile set reproduces the source byte for byte, trailing newline included.

Part 3: commit, encrypt-gate, deploy through Composer

Once the new value is in a dotenv file that gets committed, the global pre-commit hook blocks staging any .env, .tfvars, or .tfstate (or .tfstate.backup) file unless it is already SOPS- or age-encrypted. Detection is by content, never by filename: ENC[AES256_GCM,, a sops_ / sops: metadata marker, or a raw age-armoured header. A repo can opt out entirely with .allow-unencrypted, or per glob with .allow-unencrypted-paths. When the hook blocks a commit, its own suggested remediation is sops --encrypt --in-place <file>.

Composer decrypts a stack’s SOPS-encrypted .env before every up, sync, or deploy, and re-encrypts it afterward: decryptSopsSecrets -> docker compose up -> reEncryptSopsSecrets. POST /stacks/<name>/deploy?async=true runs the full pipeline - git pull, decrypt, compose pull, compose up -d, re-encrypt. Running docker compose by hand against a checkout instead hands the container literal ciphertext strings as environment values (for example POSTGRES_DB=ENC[AES256_GCM,data:...]), which fails the container’s healthcheck.

After a deploy, confirm the checkout is ciphertext again with grep -q sops_version <checkout>/.env - the marker is present only in the encrypted form. Checking the first three bytes of the file for ENC or sops gives a false “plaintext” positive, because an encrypted dotenv’s first line is simply the first key’s name.

Rotating a database role credential

For a role credential specifically - a stack whose database role password is rotating - the order matters more than for a plain application secret, because the database and the deploying stack are not updated atomically:

1. Rotate the value in the stack’s SOPS-encrypted .env, in git; commit and push.
2. ALTER ROLE / ALTER USER ... WITH PASSWORD '...' on the RUNNING database FIRST, via secretctl exec or an admin exec endpoint, so the live container keeps working through the deploy window.
3. Trigger the stack deploy (sync, decrypt, pull, up, re-encrypt).
4. Verify the checkout is ciphertext again.

This order is documented rather than exercised in this guide: no live database was touched while writing it, in keeping with the read-only rule for this kind of worked example. Reversing steps 1 and 2 - deploying before the role’s password changes - restarts the container with a value the database does not yet accept.

Part 4: prove propagation with fp and cmp

secretctl fp and secretctl cmp compare sources by keyed digest, never by value. By default the salt is ephemeral, so a digest printed in one run means nothing in the next; verifying a rotation propagated across hosts needs a PERSISTENT salt instead:

secretctl fp --salt-file salt-file <src>    # before the deploy
secretctl fp --salt-file salt-file <src>    # after the deploy, same salt file

A changed digest proves the value changed on that host between the two runs; an unchanged digest proves nothing has happened there yet - it is not evidence that the rotation failed, only that this particular host has not received it. --salt-file on a path that does not yet exist creates it (mode 0600) with a fresh random salt on first use; a path that already exists but holds fewer than 16 bytes is rejected rather than silently treated as absent - a stray touch salt-file before the first run fails with secretctl fp: salt file salt-file is too short (0 bytes, need >= 16) rather than generating a new salt in it.

Worked rotation

The full sequence, run in a scratch directory against a throwaway registry (two dotenv: stores plus one keyfile: canary - none of it a real credential):

$ secretctl cmp dotenv:.../new.env#CANARY_TOKEN dotenv:.../old.env#CANARY_TOKEN
  dotenv:.../new.env#CANARY_TOKEN  MATCH  <hmac> (48B)
  dotenv:.../old.env#CANARY_TOKEN  MATCH  <hmac> (48B)
=== MATCH: every source holds the same credential. ===
exit: 0

$ secretctl set dotenv:.../new.env#CANARY_TOKEN --from keyfile:.../canary.key
  dotenv:.../new.env#CANARY_TOKEN
    before  <hmac> (48B)
    after   <hmac> (48B)
exit: 0

$ secretctl cmp dotenv:.../new.env#CANARY_TOKEN dotenv:.../old.env#CANARY_TOKEN
  dotenv:.../new.env#CANARY_TOKEN  MATCH     <hmac> (48B)
  dotenv:.../old.env#CANARY_TOKEN  MISMATCH  <hmac> (48B)
=== MISMATCH: the credentials differ. ===
exit: 1

$ secretctl cmp dotenv:.../new.env#CANARY_TOKEN keyfile:.../canary.key
  dotenv:.../new.env#CANARY_TOKEN  MATCH  <hmac> (48B)
  keyfile:.../canary.key           MATCH  <hmac> (48B)
=== MATCH: every source holds the same credential. ===
exit: 0

The rotated store (new.env) now disagrees with the store nobody has updated yet (old.env, exit 1) and agrees with the value it was rotated from (canary.key, exit 0) - the exact pair of results a real rotation should show for “the store I changed” against “the store I have not gotten to yet” and “the value I rotated to”. The same shape holds for a sops: destination: set into a sops-encrypted dotenv decrypts, upserts the one field, and re-encrypts to the SAME recipients the file already carried - the sops_age__list_0__map_recipient line is byte-identical before and after the write, and the file stays fully ciphertext throughout, confirmed this run by grepping for ENC[AES256_GCM and sops_version after the write.

Part 5: what the guard sees, and when

The digest cache the secret-output guards read is refreshed by a systemd timer every ~10 minutes. As of 2026-09-29, in the primary harness only, a hook also forces a refresh after a command that ran secretctl set returns. That refresh waits on the cache’s lock (flock -w 60, not a non-blocking attempt) rather than skipping, because an in-flight timer pass may have already read the store before the write landed - skipping in that case would publish the old value’s digest as if it were current. If the lock is still held after 60 seconds, the forced refresh gives up, and the next timer pass - up to about 11 minutes later - picks the new value up instead.

A rotation made from outside the primary harness - a different shell, a script, something run directly on the router - has no forced refresh to wait for. It is only ever picked up by the next scheduled timer pass. The full writer-and-lock protocol behind this, including the three-writer history that predates it, is described in Keeping credentials out of coding-agent transcripts; this guide only needs the one consequence: a set you just ran may not be visible to the guard for up to a timer period, and how long that wait is depends on which harness ran the command.

Verification

Check	How	Expected
The write landed	before/after digest printed by set	after differs from before, unless the value was deliberately unchanged
The rotated store, alone	secretctl cmp <rotated> <value it was rotated from>	MATCH, exit 0
A store not yet updated	secretctl cmp <rotated> <un-rotated store>	MISMATCH, exit 1
Propagated to a remote host	secretctl fp --salt-file <file> <src>, before and after, same salt file	digest changes on that host
Checkout still SOPS-encrypted	grep -q sops_version <checkout>/.env	present
sops recipients unchanged by the rotation	grep map_recipient <file> before and after	byte-identical
Deploy pipeline completed	Composer stack status / container healthcheck	healthy, no ENC[AES256_GCM, in a running container’s env

Gotchas and lessons learned

• The dotenv newline refusal was too strict until 2026-09-29. set now canonicalises the same way fp/cmp do before checking for a newline (Part 2), so a key file with the one ordinary trailing newline that openssl rand -hex 24 > file leaves behind now sets cleanly. An interior newline, or a second trailing newline, is still refused - the fix narrows a false refusal, it does not widen what dotenv can hold.
• A freshly minted value is invisible to the guard until its store is registered. This is not a bug to work around at rotation time; it is the reason Part 1 captures by reference and registers the store before using the value a second time.
• An empty --salt-file fails closed rather than silently generating a new salt. If the path exists with fewer than 16 bytes - a stray touch, an interrupted first run - fp/cmp refuse it. Let secretctl create the file itself, or make sure it never exists as a zero-byte file first.
• The reader and writer of the dotenv format used to disagree. Before they were unified, the writer escaped a double quote as \" and the reader returned the literal backslash, so a value written by set and read back by fp reported MISMATCH on a credential that had not changed. internal/dotenv now defines both together and round-trip-tests them, which is also why the guide above trusts cmp’s MATCH/MISMATCH rather than a manual byte comparison.
• A rotation’s visibility to the guard depends on which harness ran set. Only the primary harness forces a refresh after the command returns; everything else waits for the next timer pass. Do not assume a rotation is visible immediately just because set printed exit 0.

File reference

File	What it is
internal/cli/cli.go	set dispatch: destination-writability check, before/after digest print, exit codes
internal/transfer/transfer.go	UpsertDotenv: the canonicalisation and newline refusal set applies to a dotenv-family destination
internal/transfer/sops.go	SopsAgeRecipients: reads the file’s own age recipients for a sops: destination
internal/credential/digest.go	LoadSalt: creates a persistent salt file if absent, refuses one shorter than 16 bytes
AGENTS.md	secretctl’s own design-invariants doc; the digest-cache writer protocol Part 5 draws on

Related docs

Keeping credentials out of coding-agent transcripts is the architecture this guide’s Part 1 and Part 5 depend on: the registry, the digest cache, and the writer-and-lock protocol behind the forced refresh after set. Where that reference doc explains how the guard is built, this guide is the task-sequenced how-to for the one action - rotating a credential - that exercises every layer of it.

secretctl: stores, digests and commands is the command and scheme reference behind every step here.