HomelabServices and mediareference
Deleting your own Discord messages on a schedule, with an encrypted archive
discordctl is a Go CLI and rolling-retention daemon that deletes a user’s own Discord messages older than a configurable cutoff while preserving an age-encrypted archive of every removed message, its surrounding channel context, and its attachments. It runs as a single-container Compose stack deployed through Composer with its state and archive directories on the host paths covered by declarative homelab backups.
TL;DR:
- Every delete path is gated by an archive commit (
internal/archive/writer.go). For each search page or export channel batch,discordctlwritesage(gzip(JSONL))records and age-encrypted attachment blobs,fsyncs the files and parent directories, and runsDELETEcalls only afterCommitreturnsnil. - The daemon holds only age X25519 public keys (
ARCHIVE_RECIPIENTS). The corresponding private identity stays off the host, so a compromise of the container or NAS mount exposes ciphertext names and sizes without message bodies or attachment filenames. - Rate limits are paced from response headers (
X-RateLimit-RemainingandX-RateLimit-Reset-After1), bounded below by a hard 0.3 sminDeleteDelayfloor (cmd/discordctl/config.go) and ratcheted upward for the rest of a scope whenever an HTTP429occurs. - Only the authenticated user’s messages can be deleted: search queries hardcode
author_id=<me.ID>, export files contain only the user’s own messages, and context or incoming DM messages are stored in read-onlyarchive.Recordstructs that never feed the delete loop. - Known Discord delete refusals have explicit handlers (
internal/discord/client.go): error50083(archived thread) queues the message for an unarchive-delete-rearchive drain at the end of the page, and error50021(system message) records the message ID instate.jsonundernondeletableso later passes skip it2.
Architecture
Section titled “Architecture”The data flow across the three input paths and the archive gate:
| Phase | Code | What it does |
|---|---|---|
| Export | phaseExport in cmd/discordctl/run.go | On the first pass when /data/export/Messages is present and export_consumed is false, discordctl reads the official Discord data export channel by channel, commits one encrypted segment per channel for messages older than that channel’s cutoff, and deletes those messages. |
| Live catch-up | phaseLiveCatchupAll and liveCatchup in cmd/discordctl/search.go | Enumerates every guild and open DM channel via /users/@me/guilds and /users/@me/channels, resolves per-scope retention cutoffs, and pages through Discord’s search endpoint (author_id=<me.ID>, max_id=<cutoff_snowflake>3). Each page of up to 25 hits is archived with surrounding context and attachment blobs into one segment before the page’s delete loop runs. |
| Incoming DM sweep | sweepIncoming in cmd/discordctl/incoming.go | Reads each open DM and group DM newest-first down to the per-channel snowflake stored in state.json under incoming_watermarks, archiving other participants’ messages and attachments as kind: "incoming" records without generating any delete targets. |
| Offline decryption | cmd/discordctl/archive_cmd.go | discordctl archive subcommands read /data/archive using an age identity file on a trusted machine without reading DISCORD_TOKEN or opening network connections. |
Operating modes: daemon, one-shot purge, and offline archive CLI
Section titled “Operating modes: daemon, one-shot purge, and offline archive CLI”| Mode | Subcommand | Network and token | Default retention cutoff | Primary use |
|---|---|---|---|---|
| Rolling daemon | discordctl run --watch | Requires DISCORD_TOKEN | 14 days (RETENTION_DAYS), repeatable every 24 hours (INTERVAL_HOURS) | Unattended container service sweeping all guilds and DMs on a schedule |
| Single pass | discordctl run | Requires DISCORD_TOKEN | 14 days (RETENTION_DAYS) | One-off full-account retention pass from a shell or cron job |
| Targeted wipe | discordctl purge | Requires DISCORD_TOKEN | 0 days (deletes all matching messages in scope unless --retention-days or --before is passed) | Immediate wipe of specific --guild or --channel scopes, optionally narrowed by --content, --after, and --before |
| Discovery preview | discordctl search | Requires DISCORD_TOKEN | Read-only preview | Inspects matching messages across guilds or DMs before running purge |
| Offline archive reader | discordctl archive cat | blob | verify | ingest-export | dataset | No token, no network calls | Not applicable | Decrypts segments and blobs, verifies SHA-256 digests, folds exports into the archive, or builds JSONL training sets |
Daemon lifecycle and parking
Section titled “Daemon lifecycle and parking”In compose.yaml, the service runs /discordctl run --watch with restart: unless-stopped, memory capped at 256 MiB, /data/state and /data/archive mounted read-write, and /data/export/Messages mounted read-only. Between passes, the daemon sleeps in 60 s increments (cmd/discordctl/run.go) and rewrites the heartbeat file (/data/state/heartbeat) on each tick so the container health check (["CMD", "/discordctl", "status", "--max-heartbeat-age", "36h"]) can distinguish a sleeping daemon from a wedged pass.
Discord user tokens have no refresh flow. When a condition requires operator action, park() in cmd/discordctl/run.go logs a banner, sends a high-priority webhook notification if NTFY_URL is configured, and blocks on select {} until SIGTERM so Docker does not spin in a restart loop:
- Token rejected (
401): triggered at startup, during preflightGetMe()before each pass, mid-pass during search/delete/archive calls, or by the scheduled-message outbox goroutine. - Identity changed: preflight
GetMe()before each pass verifies thatfresh.ID == me.ID; if the token now resolves to a different user ID, the daemon parks before issuing any search or delete call. - Archive misconfigured or unwritable: triggered when
ARCHIVE_RECIPIENTSis empty or fails X25519 parsing while archiving is enabled, or when/data/archivefails directory creation,chmod,fsync, orrename. - State unwritable: triggered when
/data/statecannot be created or written. - Restart burst:
state.jsontracksrestart_burstandlast_started_at; more than 5 startups within 600 s parks the container, and a successfulGetMe()resetsrestart_burstto 0.
One-shot purge and Cobra flag isolation
Section titled “One-shot purge and Cobra flag isolation”purge gotcha | Detail |
|---|---|
| Search stemming and date bounds | discordctl purge shares the liveCatchup engine and state.json format with discordctl run, but scopes its work to explicit --guild or --channel flags. Because Discord’s search API stems query words, purge --content <phrase> sends content to the search endpoint and then re-checks every returned hit locally via case-insensitive substring match in ScopedTarget.selects (cmd/discordctl/search.go), skipping non-matching hits without marking them deleted. Date bounds (--after and --before) are parsed strictly as YYYY-MM-DD, RFC3339, or RFC3339Nano; --before can only narrow the retention cutoff (min(beforeSF, cutoffSF)), and --after >= cutoff exits with code 2. |
Shared flag defaults and Go init() order | Both run and purge bind shared package-level flag variables (retentionDays, deleteDelay, searchDelay, statePath, and the archive* flags). Go executes init() functions in filename order, so search.go registers purge’s default retentionDays = 0 after run.go registers 14. To prevent run from inheriting 0 when --retention-days is omitted on the command line, both commands call resolveFloat, resolveString, resolveBool, and resolveSlice at the start of Run (cmd/discordctl/run.go, cmd/discordctl/archiver.go), checking cmd.Flags().Changed(name) before falling back to environment variables and per-command defaults. |
Archive-before-delete pipeline
Section titled “Archive-before-delete pipeline”Archiving is enabled by default and fails closed (internal/archive/config.go). Unless --no-archive (ARCHIVE_DISABLED=1) is explicitly set, LoadConfig requires at least one valid X25519 public key in --archive-recipient (ARCHIVE_RECIPIENTS)4. A malformed recipient string returns an error immediately.
Live catch-up page flow
Section titled “Live catch-up page flow”For each page of search hits in liveCatchup (cmd/discordctl/search.go), arch.archivePage (cmd/discordctl/archiver.go) builds one archive.Record per hit before any DELETE request is sent for that page:
- Anchor record construction: stores the raw JSON message object from the search hit (
h.Raw), the resolved guild and channel metadata (GuildRefandChannelRef),kind: "anchor",source: "live", and the UTCarchived_attimestamp. - Context window fetch (
FetchAround): whenARCHIVE_CONTEXTis greater than 0 (default 100, clamped to Discord’s 1..100 limit because 101 returns HTTP400code500355), the archiver callsGET /channels/{id}/messages?around=<msgID>&limit=100. To avoid redundant fetches when several deleted messages sit near each other in the same channel,a.windows[channelID]caches the[minID, maxID]snowflake bounds of every successfulFetchAroundin the current pass. If a later anchor ID falls within an existing window, the network call is skipped and the record setscontext_covered: true. - Attachment streaming (
PutBlobStream): attachments on the anchor message and on context messages are fetched from Discord’s CDN using a separatediscord.Downloader(internal/discord/client.go) that never attaches theAuthorizationheader to signed CDN URLs. Bytes stream throughio.MultiWriter(enc, sha256)directly intoblobs/.<attachment_id>.age.tmp, keeping memory bounded to the age chunk size (avoiding 256 MiB container OOM kills on large video attachments), thenfsync, rename toblobs/<attachment_id>.age, andfsynctheblobs/directory (internal/archive/writer.go). Because Discord attachment snowflakes are globally unique,HasBlobskips any attachment ID whose.agefile already exists on disk. - Segment commit (
Writer.Commit): all non-deferred records for the page are marshalled as JSON Lines, compressed withgzip.BestCompression, encrypted toARCHIVE_RECIPIENTS, and written tosegments/YYYY/MM/<unixnano>-<8_hex_rand>.jsonl.gz.agevia temporary filefsync, atomic rename, and directoryfsync(including parentYYYY/andMM/directories).
Three-layer only-my-messages invariant
Section titled “Three-layer only-my-messages invariant”discordctl enforces at three boundaries that it never attempts to delete another user’s message:
- Search query construction (
SearchMessagesininternal/discord/client.go): every/guilds/{id}/messages/searchand/channels/{id}/messages/searchcall setsauthor_idto the authenticated user’s ID returned by/users/@me. - Type separation in the archiver (
cmd/discordctl/archiver.goandcmd/discordctl/incoming.go):FetchAroundandFetchPagereturn[]json.RawMessagecontaining messages from all channel participants. Those slices are placed exclusively intoarchive.Record.Contextorkind: "incoming"records; onlydiscord.SearchResultslices fromSearchMessagesandexport.Messageslices from the user’s export directory reachDeleteMessage. - Non-retryable
403on delete (DeleteMessageininternal/discord/client.go): if aDELETEcall ever returns HTTP403(without error code50083or50021), it maps to"forbidden", incrementsdiscord_wipe_deletes_total{outcome="forbidden"}, marks the ID instate.json, and never retries within the pass.
Rate-bucket pacing and retry floors
Section titled “Rate-bucket pacing and retry floors”Discord enforces per-route rate limits communicated through response headers and global rate limits communicated via HTTP 429 payloads1. discordctl combines header-driven pacing with hard minimum delays and per-scope 429 floors (internal/discord/client.go, cmd/discordctl/config.go, cmd/discordctl/search.go).
Header-driven bucketPacing
Section titled “Header-driven bucketPacing”On every 204 or 404 from DeleteMessage, 200 from SetThreadArchived, and 200 from FetchAround/FetchPage, bucketPacing(resp) inspects X-RateLimit-Remaining (rem) and X-RateLimit-Reset-After (reset, in seconds):
| Header state | Returned pacing hint (RetryAfter) | Behaviour |
|---|---|---|
| Headers absent or unparseable | 0 s | Caller uses configured deleteDelay (default 1.0 s) |
rem == 0 and reset > 0 | reset | Bucket exhausted on this call; wait the full reset window before next request |
rem > 0 and reset > 0 | reset / float64(rem) | Spreads remaining requests evenly across the bucket’s remaining window |
floor429 ratcheting and hard floors
Section titled “floor429 ratcheting and hard floors”Within each target scope in liveCatchup (and each channel in phaseExport), the delete loop maintains two floating-point timers alongside the configured deleteDelay (DELETE_DELAY, default 1.0 s): extraSleep and floor429.
- When
DeleteMessagereturns"retry"on HTTP429,parseRetryAfterreadsretry_afterfrom the JSON response body (falling back to theRetry-Afterheader, or 1 s if both are zero). The loop setsextraSleep = result.RetryAfterandfloor429 = result.RetryAfter, sleeps forresult.RetryAfter, and retries the same message ID. - On subsequent
"ok"or"gone"responses in the same scope, ifresult.RetryAfter > 0,extraSleepupdates tomax(result.RetryAfter, floor429). Once a scope hits a429, inter-delete sleep for that scope never drops belowfloor429even if later bucket headers advertise a shorterreset / remratio. - After each message is processed, the loop sleeps
max(deleteDelay, extraSleep). requireDeleteDelayFloor()incmd/discordctl/config.goenforcesminDeleteDelay = 0.3s acrossrunandpurge, exiting with code 2 if--delete-delayorDELETE_DELAYis set below 0.3 s. Policy files (config.toml) also enforceminSearchDelay = 1.0s (defaultSEARCH_DELAY = 15.0s).- Transient transport errors inside
Client.doBodyretry up toDefaultRetry = 8attempts with exponential backoff2.0 * 2^(attempt - 1)s capped atRetryCap = 30.0s (internal/discord/client.go). Search endpoints treat HTTP202(search index building),429,5xx, and JSON decode errors as unknown states with a retry hint; 8 consecutive search retries (retryStreak >= 8) skip the scope for the current pass without marking any messages deleted.
Retention overrides and channel carve-outs
Section titled “Retention overrides and channel carve-outs”In discordctl run, retention cutoffs can be tuned per guild, channel, or channel category:
| Flag | Environment variable | Purpose |
|---|---|---|
--retention-override | RETENTION_OVERRIDES | Per-guild, per-channel or per-category retention cutoff |
--exclude-guild | EXCLUDE_GUILDS | Skip a whole guild |
--exclude-channel | EXCLUDE_CHANNELS | Skip a channel |
| TOML policy file | $XDG_CONFIG_HOME/discordctl/config.toml or DISCORDCTL_CONFIG | The same policy in a file, parsed in cmd/discordctl/config.go and cmd/discordctl/overrides.go |
Override grammar and precedence
Section titled “Override grammar and precedence”Each override entry uses the 3-part syntax <scope>:<id>:<days>, where <days> is a non-negative float (0 deletes all messages in that scope on every pass):
| Precedence (highest to lowest) | Rule syntax | Scope matched |
|---|---|---|
| 1 | --exclude-guild <id> / --exclude-channel <id> (or exclude = true in TOML) | Skips the guild, server channel (plus its threads), DM, or group DM completely |
| 2 | channel:<snowflake>:<days> | Specific server channel (and its threads), DM, or group DM |
| 3 | guild:<snowflake>:<days> | Specific guild |
| 4 | kind:dm:<days> / kind:groupdm:<days> / kind:discoverable:<days> | All 1-on-1 DMs (Type == 1), group DMs (Type == 3), or guilds whose features array includes "DISCOVERABLE" |
| 5 | --retention-days <days> (RETENTION_DAYS, default 14) | Global fallback across all remaining guilds and DMs |
Any malformed entry in RETENTION_OVERRIDES or unknown key in config.toml terminates discordctl run with exit code 2 before any API call is made, preventing a typo from falling through to a shorter retention window.
Server-channel carve-outs and thread ownership
Section titled “Server-channel carve-outs and thread ownership”Discord’s guild search endpoint (/guilds/{id}/messages/search) searches the entire guild unless channel_id is passed, and a search with channel_id=<parent> returns hits from both the parent channel and its child threads. When a server channel has an override or exclusion, phaseLiveCatchupAll (cmd/discordctl/run.go and cmd/discordctl/overrides.go) resolves each configured channel ID via GET /channels/{id} (resolveChannelGuilds) to find its guild_id:
- If
GET /channels/{id}returns any error other than403or404(or terminal401),phaseLiveCatchupAllskips all guild targets for that pass so a transient failure cannot cause a protected channel to be swept under its parent guild’s shorter cutoff. buildRunTargetsplaces every carved channel ID for a guild into a sharedSkipChannelsmap on both the guild-wideScopedTarget(ChannelID: "") and each carved channel’s dedicatedScopedTarget(ChannelID: cid).- When a search response arrives,
SearchMessagesmaps top-levelthreads[].id -> threads[].parent_idonto eachSearchResult.ParentID(internal/discord/client.go).ScopedTarget.selects(cmd/discordctl/search.go) assigns each hit a single owner:m.ChannelIDif that ID is inSkipChannels, elsem.ParentIDif the thread’s parent channel is inSkipChannels, else""(the guild-wide target). A target processes only hits whose computed owner equalst.ChannelID. Excluded channels are placed inSkipChannelswithout generating a channel target, so no target selects their messages or threads. - In
phaseExport, export rows lack aparent_idfield for threads. Whenever any server-channel override or exclusion is active (deferThreads == trueincmd/discordctl/run.goandexportChannelDaysincmd/discordctl/overrides.go), the export phase skips all*_THREADexport folders and leaves them toliveCatchup, which has thread parent metadata from the search API.
Encrypted archive format and offline decryption
Section titled “Encrypted archive format and offline decryption”The archive directory (/data/archive, mode 0700) contains two subdirectories (segments/ and blobs/, both mode 0700) and stores no plaintext files (internal/archive/writer.go, internal/archive/record.go):
/data/archive/├── segments/│ └── 2026/│ └── 10/│ └── 1760227200000000000-a1b2c3d4.jsonl.gz.age└── blobs/ └── 111111111111111111.age- Segments:
segments/YYYY/MM/<unixnano>-<8_hex_rand>.jsonl.gz.age(mode0600). Decrypting withageand decompressing withgzipyields UTF-8 JSON Lines, onearchive.Recordper line. Because filenames start withtime.Now().UnixNano(), lexicographical path order matches chronological commit order. - Blobs:
blobs/<attachment_id>.age(mode0600), holding the raw attachment bytes encrypted toARCHIVE_RECIPIENTS. Filenames on disk use only the numeric Discord attachment ID (validAttachmentIDrejects non-digit characters as a path-traversal guard); the originalfilename,content_type,size, and plaintextsha256hex digest are stored inside theRecord.Blobsarray in the segment JSONL.
Each JSONL line in a segment has the archive.Record schema (internal/archive/record.go):
{ "v": 1, "kind": "anchor", "source": "live", "archived_at": "2026-10-12T00:00:00Z", "guild": { "id": "111111111111111111", "name": "Example Guild" }, "channel": { "id": "222222222222222222", "name": "general", "type": 0, "parent_id": "" }, "message": { "id": "333333333333333333", "channel_id": "222222222222222222", "content": "example message" }, "context": [], "context_covered": false, "context_error": "", "blobs": [ { "attachment_id": "444444444444444444", "message_id": "333333333333333333", "filename": "diagram.png", "content_type": "image/png", "size": 18432, "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "error": "" } ]}Decrypting and verifying a read-only copy
Section titled “Decrypting and verifying a read-only copy”To inspect or process the archive on a machine holding the age private key file (keys.txt), mount or sync /data/archive read-only and run the token-less discordctl archive subcommands (cmd/discordctl/archive_cmd.go):
# Stream all records in chronological order as JSONL to stdoutdiscordctl archive cat \ --identity /run/secrets/discord-archive-key.txt \ --archive-dir /data/archive
# Stream only segments committed at or after an RFC3339 timestamp# (skips older segments by filename prefix before decryption, so a key added# to ARCHIVE_RECIPIENTS at that timestamp can read newer segments cleanly)discordctl archive cat \ --identity /run/secrets/discord-ingest-key.txt \ --archive-dir /data/archive \ --since 2026-10-15T00:00:00Z
# Stream a single decrypted attachment blob to stdout (never writes plaintext to disk;# exits 0 on success, 3 if missing, 4 if the identity cannot decrypt the blob)discordctl archive blob 444444444444444444 \ --identity /run/secrets/discord-archive-key.txt \ --archive-dir /data/archive > diagram.png
# Decrypt every segment and referenced blob, recompute SHA-256 digests, and report countsdiscordctl archive verify \ --identity /run/secrets/discord-archive-key.txt \ --archive-dir /data/archive
# Sessionise decrypted anchors and context into an LLM fine-tuning JSONL datasetdiscordctl archive dataset \ --identity /run/secrets/discord-archive-key.txt \ --archive-dir /data/archive \ --gap 1h --min-turns 2 \ --out /tmp/discord-sessions.jsonlKnown Discord delete errors and pagination guards
Section titled “Known Discord delete errors and pagination guards”Discord returns specific JSON error codes inside HTTP 400 and 403 bodies when a message cannot be deleted in its current state2. DeleteMessage (internal/discord/client.go) parses {message, code} from the response body and maps these codes to dedicated status strings so liveCatchup and phaseExport can handle them without stalling pagination.
HTTP status and JSON code | DeleteResult.Status | Meaning in Discord API | discordctl action |
|---|---|---|---|
204 No Content | "ok" | Message deleted | Marks ID in state.Deleted, increments ok counter, paces via bucketPacing |
404 Not Found | "gone" | Message already deleted | Marks ID in state.Deleted, increments gone counter, paces via bucketPacing |
400 or 403 with code: 50083 | "archived" | Thread is archived - messages inside an archived thread cannot be deleted until the thread is unarchived2 | Queues msgID in pendingArchived[channelID] without marking it deleted; at the end of the page or channel batch, drainArchivedThreads calls PATCH /channels/{id} with {"archived": false}6, deletes the queued messages, marks them in state.Deleted, and calls PATCH /channels/{id} with {"archived": true} |
400 or 403 with code: 50021 | "nondeletable" | Cannot execute action on a system message (recipient pins, call notifications, boosts, thread-created notices)2 | Calls st.Mark(id) and st.MarkNondeletable(id) so the ID is persisted in state.json under both deleted and nondeletable, then skips it on all future passes |
400 or 403 (any other code) | "forbidden" | Missing permissions or locked resource | Marks ID in state.Deleted for the current state, increments forbidden counter; if a whole page of marked messages still appears in search on a later pass, liveCatchup re-attempts non-50021 messages at most once per pass (retriedThisPass) |
401 Unauthorized | "auth" (*AuthError) | User token rejected or rotated | Saves state.json immediately and parks the daemon (or exits 2 in purge) |
Keyset pagination past stuck messages (advanceMaxID)
Section titled “Keyset pagination past stuck messages (advanceMaxID)”Discord’s search endpoint returns 25 hits per page, ordered newest-first. When a scope contains more than 25 permanently undeletable messages (such as 50021 system messages, locked threads where PATCH /channels/{id} returns 403, or hits deferred by an attachment download failure), a naive loop that always queries from cutoffSF would fetch the same 25 stuck messages on every page and never reach older deletable messages behind them.
liveCatchup (cmd/discordctl/search.go) avoids this trap with keyset pagination:
- Each scope starts with
maxID = t.CutoffSF. - When a page has zero new deletable hits (all hits are filtered out, deferred, marked
nondeletable, or already retried once this pass) or when a page’s work was entirely deferred by the archiver (!issuedDelete),advanceMaxIDsetsmaxIDto the minimum message snowflake on the page (minHitID(hits)). Because Discord’smax_idquery parameter is exclusive, the next search query fetches messages strictly older than the oldest stuck message on the current page. - If
drainArchivedThreadscannot unarchive a thread (SetThreadArchivedreturns"forbidden"or"gone"), the queued message IDs are left unmarked instate.jsonso a future pass can retry if permissions change, and are added tosweepDeferredfor the remainder of the current pass soadvanceMaxIDsteps past them instead of re-querying the same page indefinitely.
Decision guide
Section titled “Decision guide”The same decision paths as a numbered list:
- Scheduled account-wide retention with one cutoff: run
discordctl run --watchwithRETENTION_DAYSandARCHIVE_RECIPIENTSset to your age public key. - Scheduled retention with per-guild, per-channel, or DM exceptions: run
discordctl run --watchwithconfig.tomlorRETENTION_OVERRIDES(guild:<id>:<days>,channel:<id>:<days>,kind:dm:<days>,kind:groupdm:<days>,kind:discoverable:<days>) andEXCLUDE_GUILDS/EXCLUDE_CHANNELS. - Immediate targeted wipe of one server, channel, or phrase: preview matches with
discordctl search --guild <id> --content <phrase>, then executediscordctl purge --guild <id> --content <phrase>, which archives every matched message before deleting it at a 0-day default cutoff. - Reading or verifying archived messages and attachments: sync
/data/archiveto a machine holding the age private identity and rundiscordctl archive cat,discordctl archive blob,discordctl archive verify, ordiscordctl archive datasetoffline withoutDISCORD_TOKEN.
Evidence
Section titled “Evidence”| Claim or behaviour | Status | How it was checked |
|---|---|---|
Archive commit failure prevents any DELETE call for that page (liveCatchup) or channel (phaseExport) | Tested | TestArchiveCommitFailureZeroDeletes and TestExportPhaseArchiveCommitFailureNoDeletes in cmd/discordctl/archiver_test.go |
Context messages from FetchAround and incoming DM messages never become delete targets | Tested | TestArchiveInvariantOnlySearchHitsDeleted in cmd/discordctl/archiver_test.go; TestIncomingSweepClientCannotDelete in cmd/discordctl/incoming_test.go |
| Retry-exhausted anchor attachment defers the hit without deleting or marking it; failed context attachment records blob error and still deletes anchor | Tested | TestArchiveOwnAttachmentRetryExhaustedDefersHit, TestArchiveOwnAttachmentStoreFailureZeroDeletes, and TestArchiveContextAttachmentStoreFailureStillDeletes in cmd/discordctl/archiver_test.go |
Partial CDN download stream is discarded on retry and only the clean attempt lands in .age blob | Tested | TestBug8RetryAfterPartialStreamStoresOnlyGoodBytes in cmd/discordctl/archiver_test.go; TestDownloaderNeverSendsAuthorization and TestDownloaderTooLargeOnStream in internal/discord/client_archive_test.go |
Segments and blobs contain no plaintext on disk, enforce 0700/0600 modes, and reject non-numeric attachment IDs | Tested | TestSegmentNotPlaintext, TestBlobNotPlaintextAndModes, and TestPutBlobRejectsPathTraversal in internal/archive/archive_test.go |
bucketPacing computes reset when remaining == 0 and reset / remaining when remaining > 0 | Tested | TestBucketPacing and TestDelete429ReturnsRetryAfter in internal/discord/client_test.go |
Error 50083 queues archived thread messages for SetThreadArchived(false) -> delete -> SetThreadArchived(true), and skips unmarked when unarchive returns 403 | Tested | TestDeleteArchivedThreadCodeIsArchived in internal/discord/client_test.go; TestExportPhaseArchivedDrainSuccess and TestExportPhaseArchivedCannotUnarchive in cmd/discordctl/export_archived_test.go |
Error 50021 on HTTP 400 or 403 marks the message nondeletable in state.json and skips it on subsequent passes and all-done page retries | Tested | TestDelete400SystemMessageCodeIsNondeletable and TestDelete403SystemMessageCodeIsNondeletable in internal/discord/client_test.go; TestNondeletableNotReattemptedSecondPass and TestNondeletableMixedAllDonePage in cmd/discordctl/catchup_test.go |
Server-channel overrides and exclusions carve the channel and its threads out of guild search targets, and defer *_THREAD rows in the export phase | Tested | TestBuildRunTargets and TestExportChannelDays in cmd/discordctl/overrides_test.go; TestExportPhaseDefersThreadsUnderChannelExclude in cmd/discordctl/archiver_test.go |
archive cat --since filters segment filenames by unixnano prefix before decryption so a newer ingest key can read post-rotation segments | Tested | TestOpenSegmentsSinceSkipsOldSegments and TestOpenSegmentsSinceUndecryptableInWindowErrors in internal/archive/reader_since_test.go; TestArchiveCatSincePrintsOnlyNewerRecords in cmd/discordctl/archive_cmd_test.go |
minDeleteDelay = 0.3 s floor avoids account-level anti-abuse flags on sustained bulk deletes | Design-only | Enforced in requireDeleteDelayFloor() (cmd/discordctl/config.go); Discord does not publish user-account anti-abuse thresholds |
Gotchas and lessons learned
Section titled “Gotchas and lessons learned”| Gotcha | Detail |
|---|---|
Dry-run state persistence (Bug15) | in an early version, --dry-run called st.Mark(id) and persisted state.json at the end of the pass, causing the next real pass to skip every message the dry run had visited. State.SetDryRun(true) (internal/state/state.go) keeps Mark() in memory for single-pass deduplication while Save() touches only the /data/state/heartbeat file and leaves state.json and state.json.bak untouched. |
Unarchived thread drain infinite loop (Bug20) | when a thread is locked or the user lacks permission to unarchive it, SetThreadArchived(chID, false) returns "forbidden" and drainArchivedThreads leaves the message IDs unmarked in state.json. Without adding those IDs to sweepDeferred for the rest of the current sweep (cmd/discordctl/search.go), liveCatchup saw them as new work on the next iteration and re-queried the same keyset page forever. |
Deferred-only page keyset stall (Bug19) | when every new hit on a search page had its anchor attachment fail download retries, archivePage deferred all of them and the delete loop issued zero DELETE calls. Because the messages still existed on Discord, the next search query with the same max_id returned the identical page. Tracking issuedDelete and calling advanceMaxID whenever !issuedDelete advances the keyset bound past the deferred page so older messages behind it are still swept. |
| Newly created year/month directory durability | os.MkdirAll creates segments/YYYY/MM, and writeFile fsyncs the segment file and MM/. On a crash immediately after a month rollover, the new YYYY/ or MM/ directory entry itself could be lost unless its parent directory is also synced; Writer.Commit (internal/archive/writer.go) explicitly calls syncDir on both filepath.Dir(segDir) and filepath.Dir(filepath.Dir(segDir)). |
Key rotation with archive cat --since | OpenSegments fails hard if any segment cannot be decrypted by the provided identity file. When adding a secondary age recipient (such as a dedicated ingest key for downstream indexing) to ARCHIVE_RECIPIENTS on an existing archive, pass --since <RFC3339> to discordctl archive cat; OpenSegmentsSince (internal/archive/reader.go) filters segment paths by their <unixnano> filename prefix before calling age.Decrypt. |
Related docs
Section titled “Related docs”- Declarative backups on a ZFS homelab: nix timers, sanoid, syncoid - how
/appdatastate and/tank/appdataarchive datasets are snapshotted bysanoidand replicated off-pool bysyncoid. - Appdata tiering on ZFS - why small synchronous state (
state.json) sits on the NVMe tier while write-once encrypted archive segments and blobs fit the bulk tier. - GitOps for Docker Compose with Composer - how the
discordctlCompose stack is deployed and SOPS-decrypted onto the NAS host.
References
Section titled “References”-
Discord, “Rate Limits,” Discord Developer Documentation. https://github.com/discord/discord-api-docs/blob/HEAD/developers/topics/rate-limits.mdx ↩ ↩2
-
Discord, “Opcodes and Status Codes - JSON Error Codes,” Discord Developer Documentation. https://github.com/discord/discord-api-docs/blob/HEAD/developers/topics/opcodes-and-status-codes.mdx ↩ ↩2 ↩3 ↩4
-
Discord, “API Reference - Snowflakes,” Discord Developer Documentation. https://github.com/discord/discord-api-docs/blob/HEAD/developers/reference.mdx ↩
-
Filippo Valsorda, “age - simple, modern and secure file encryption,” GitHub. https://github.com/FiloSottile/age/blob/HEAD/README.md ↩
-
Discord, “Message Resource,” Discord Developer Documentation. https://github.com/discord/discord-api-docs/blob/HEAD/developers/resources/message.mdx ↩
-
Discord, “Channels Resource - Modify Channel,” Discord Developer Documentation. https://github.com/discord/discord-api-docs/blob/HEAD/developers/resources/channel.mdx ↩