A self-hosted transcription service for coding agents

A self-hosted transcription service that runs on one GPU, shares it with an LLM engine, and surfaces compressed results to a coding agent through tools that never return raw transcripts. The agent gets a summary, a bundle path, and a set of analysis tools that compute from the same cached bundle instantly; a recorded conversation becomes a reviewable document without the agent’s context being filled with diarized text.

Provenance: every number, model name and default value below is read from the whisper-transcribe service source, the agent-side video-review-core.ts library, and the whisper skill, as of 2026-10-20. Where a figure is a default in code rather than a measured result, it is called out as a default. The overlap analysis and metrics are pure math over diarized transcripts; the frame descriptions are from a VLM. Conversation-metrics accuracy is bounded by the heuristics’ documented limits.

This stack is the transcription half of a larger workstation setup. The GPU scheduling and what a model swap costs are covered in Scheduling one GPU between LLM, image and training work; Which model runs which agent work covers the vision-fallback path that also uses the whisper stack.

TL;DR:

Two Python services in a Compose stack: whisper-transcribe (GPU-backed transcription, diarization, frame description and OCR on port 7860) and whisper-live (live transcription over a WebSocket).
The default model is turbo; five smaller/larger models are available. Translation policy defaults to "auto", which runs a 30-second language-ID pre-pass and translates non-English audio to English.
Diarization produces word-level speaker timing, which feeds every conversation-metrics tool. It triples the extract time: a 60-minute recording takes 2-3 minutes without it, 8-10 minutes with it on a consumer GPU.
Voice prints enrol a speaker’s 256-dimensional embedding from a previous diarized run and auto-name that speaker in every later transcript from the same service.
The agent never sees a raw transcript. video_extract returns a compact summary (speaker shares, duration, a mic-routing warning, and a format-tag suggestion) plus a bundle path. The bundle is cached to disk under XDG_DATA_HOME, so re-analysis is instant.
Three analysis tools run from the same cached bundle: video_overlap (acoustic collisions, who came in over whom, who yielded), video_metrics (speaking rate, fillers, turn-gap percentiles, question-flow classification), and video_doc (a markdown evidence bundle for the agent to turn into a review doc).
The conversation metrics are bounded by their heuristics. Overlap cannot distinguish a steering question from an information-seeking one; question-flow classification (DAMSL-lite, form-based) is ~70% accurate; speaker-labelling from address cues is a suggestion, never a verdict.
A shared GPU means transcription evicts the LLM engine. After a job, the LLM container restarts and the next agent turn pays a cold-context prefill penalty: 40-90 seconds to first token at about 200k tokens of history.

Topology

Text fallback:

The agent harness calls six tools (video_extract, video_overlap, video_metrics, video_doc, video_enroll, video_name) that call the whisper service’s HTTP API on port 7860. On pi these are extension tools; on Claude Code they are exposed through a toolkit MCP server that wraps the same core library.
The whisper stack is two services plus a job queue: whisper handles transcription, diarization, VLM frame description, OCR and voice prints; whisper-live does streaming; valkey is the FIFO queue that serialises all consumers (bot, MCP, UI, curl).
The GPU runs either the LLM engine or whisper, never both. A transcription job stops the LLM container; the next chat request restarts it.
The agent also sends LLM requests to the GPU proxy (port 11434). The proxy owns container lifecycle and mode-switching; it is covered in the GPU scheduling page.

Which tool for which need

You need	Use	Returns	Notes
Transcript of a recording, plus a cached bundle for re-analysis	`video_extract`	Summary + bundle path	Slow (minutes for a long recording). The agent never receives the raw transcript; the bundle is cached to disk.
Who talked over whom, and what the collisions say about the conversation	`video_overlap`	Per-speaker overlap stats, event list, pair-cluster matrix	Instant from cache. Minimum collision threshold 0.3 s (default).
Speaking style: rate, fillers, turn-gap percentiles, question-flow classification	`video_metrics`	Per-speaker metrics, per-pair gap matrix, question-outcome breakdown, block analysis	Instant from cache. Heuristic; all classifications are ~70% accurate.
A markdown evidence bundle to write a review document from	`video_doc`	Transcript + visual timeline + overlap summary + speaker table	Instant from cache. The agent writes the doc from this, not the service.
Auto-name a speaker across all future transcripts from the same service	`video_enroll`	Enrolment confirmation	Stores a 256-d voice print server-side.
Correct speaker labels in one cached bundle	`video_name`	Updated bundle	Client-side only; does not enrol a voice print.

All analysis tools (video_overlap, video_metrics, video_doc) work from a cached bundle and complete in under a second. Only video_extract runs the GPU pipeline; it is the bottleneck.

The transcription pipeline

video_extract orchestrates three steps, each optional beyond the first:

Transcription + diarization. A POST /api/jobs enqueues on the valkey-backed FIFO with the chosen model, language hint, translation policy, and diarization flag. The tool polls GET /api/jobs/<id> every 4 seconds until the job reaches done, failed, or a configurable timeout (default 1800 seconds). On success it receives word-level segments with speaker labels and, when diarization is on, per-speaker 256-dimensional WeSpeaker embeddings.
Frame description (optional). When frames: true, a POST /api/describe extracts frames at a configurable interval (default 10 seconds, cap 60 frames) and sends each to a VLM for a one-sentence description. This is the visual counterpart to the transcript - useful when a recording includes screen shares, slides or a whiteboard.
Bundling. The segments, optional frames, speaker list, embeddings, and parameters are written as a JSON bundle under XDG_DATA_HOME/video-review/<hash>.json. The cache key is a truncated SHA-256 of the server-side file path and the extract parameters, so the same recording extracted with different settings (e.g. with and without diarization) produces separate bundles. A lightweight index.json beside the bundles enables cross-bundle lookups without parsing every file.

The extract summary the agent sees is deliberately compact:

File name and recording duration.
Active speech share: clocked speaking time as a percentage of wall time.
Per-speaker speaking shares, sorted longest first.
A warning when the enrolled owner is near-silent (possible mic-routing failure).
A format-tag suggestion when the bundle is untagged: heuristics classify a 2-speaker named call as a pair conversation, a call with one dominant speaker as a presentation format, and a 3+ speaker call with at least half the speakers named as a team call. The tag is a free-form label stored on the bundle; video_metrics then builds per-format longitudinal baselines from it.
The bundle path, so the agent can pass it to the analysis tools.

The agent never receives the raw transcript in its context. The bundle stays on disk; the tools read it and return compressed results.

Job handling and polling

The /api/jobs endpoint returns 202 Accepted with a job_id immediately. The tool polls every 4 seconds until completion or timeout. This matters because the agent’s bash tool times out at 30 seconds while transcription of a long recording takes minutes. The poll loop runs inside the agent’s tool execution, so the call blocks but does not re-prompt the model.

Long recordings are the normal case. A 60-minute call on turbo with diarization takes 8-10 minutes on an RTX 5090 (observed; not a measured benchmark). Without diarization it takes 2-3 minutes. The time is dominated by the diarization pass, which runs a separate neural model over the audio; transcription alone is cheaper.

Translation policy and the language-ID pre-pass

The translate parameter defaults to "auto", which runs a 30-second language-ID pre-pass over the audio before transcription. If the detected language is not English, the transcription runs with task=translate, producing an English transcript from non-English audio. If the language is English, no translation runs. Setting translate: true forces translation regardless; translate: false keeps the source language.

This default exists because the downstream summarisation LLM reads English best. The pre-pass costs about 30 seconds but avoids translating audio that is already in English and avoids the degradation that forced translation sometimes produces on English input.

There is a known limitation: Whisper’s per-segment decoder occasionally ignores the translate task token on deliberately multilingual audio (several short segments, each in a different language). The model’s high-confidence per-chunk language detection can override the translate flag. The practical effect is that some non-English segments survive untranslated in the transcript. The downstream LLM usually reads them natively and produces a coherent English summary regardless, so the chain works end to end for summarisation use cases. For verbatim-transcript use cases where every word must be English, a post-processing LLM translation step would be needed; the service documents this as deferred until concrete demand surfaces.

Diarization

Diarization assigns each word to a speaker (SPEAKER_00, SPEAKER_01, …). It is optional but required for every analysis tool: overlap detection, per-speaker metrics, question-flow attribution, and block analysis all need word-level speaker timing. Without diarization the bundle stores segments only, with no speaker labels and no word timing, so the analysis tools return an error.

The trade-off is speed. Diarization roughly triples the extract time. A recording that transcribes in 2 minutes takes 8-10 minutes with diarization on. For a recording you only need a transcript from, skip diarization. For one you intend to review, enable it.

Voice prints

A voice print is a 256-dimensional speaker embedding stored server-side and keyed by name. Once enrolled, the speaker is auto-named in every later transcript the service produces - through the agent tools, the web UI, the Discord bot and plain curl.

Enrolment works from an existing diarized bundle: video_enroll with action: "add", the bundle path, and the speaker label to pull the embedding from. The label is the diarized tag (M-SPEAKER_01), not the display name. The embedding already exists in the bundle (the server returns it from the transcription job), so enrolment needs no re-run of the audio. Enrolling the same name twice adds a second reference vector and improves matching.

Names flow into bundles through two paths:

Server-side matching. When a transcription job finishes, the server compares each diarized speaker’s embedding against all enrolled voice prints. Matches above a cosine-similarity threshold populate speaker_names in the job result, and the cached bundle stores them under bundle.names.
Client-side relabeling. video_name takes a {label: name} map and rewrites segment speakers, word speakers and the speaker list in the bundle. The map can be manual or LLM-inferred from address cues in the transcript (“Hi, Alice” -> the next different speaker is probably Alice). The extension’s suggestNames() function extracts these cues with confidence markers (high/medium/low), but they are suggestions only: a greeting pile-in where several speakers greet at once cannot attribute a responder to a specific greeting, and the function emits those at low confidence. Embedding keys are deliberately not renamed, so video_enroll can always pull a vector by the stable diarized label.

A bundle keeps the original diarized labels as stable ids and a names record that maps them to display names. Re-renaming normalises in place: renaming a label that already maps to a name updates the existing entry rather than chaining it.

Conversation analysis from word-level timing

Diarization produces per-word speaker labels with start and end timestamps. The analysis tools use these, not segment-level timing, because segment boundaries are rounded to utterance edges and misrepresent actual acoustic collisions. Two speakers can overlap within what a segment-level view calls sequential turns; word-level timing catches the real overlap.

Per-speaker utterance merging

Before any analysis, words are grouped by speaker and merged into utterances: consecutive same-speaker words separated by no more than 0.6 seconds become one utterance. The merge runs per-speaker rather than over a globally time-sorted stream. A global sort across two overlapping speakers would interleave their words and fragment both into spurious back-and-forth utterances, inventing ping-pong overlaps that never happened. Per-speaker merging keeps each speaker’s spans whole; overlap is then a clean interval intersection between different speakers.

Speech overlaps

computeOverlap() scans time-sorted utterances for any later utterance whose start falls inside an earlier one from a different speaker. For each collision it records: who came in over whom (interrupter / interruptee), the duration of the collision, and who stopped first (yielded: the interruptee stopping means a classic talk-over; the interrupter stopping means a false start or a short backchannel).

The min_overlap_sec threshold defaults to 0.30 seconds and filters out alignment jitter and short acknowledgements (“yeah”, “mm”). The analysis returns per-speaker aggregate counts (times this speaker started over someone, times someone started over them), an event list sorted by time, and a pair-cluster matrix: for each speaker pair, how many times the first came in over the second and the total collision seconds.

The tool cannot judge intent. A steering question that interrupts someone and an information-seeking question that overlaps a closing sentence produce identical acoustic patterns. The numbers describe what happened physically; the human reviewer decides what the pattern means.

Speaking-style metrics

computeMetrics() builds on the overlap analysis and adds delivery statistics computed from utterances:

Words per minute and words per turn, from clocked speaking time and utterance counts.
Filler rate per 100 words, matched against a lexicon of filler and hedge phrases (um, uh, you know, sort of, kind of, basically, actually, I mean, to some extent, effectively).
Turn-gap percentiles (p25, median, p75): the gap between the previous different speaker’s utterance end and this speaker’s start. Negative gaps mean the speaker entered before the previous speaker finished - the fast-response tail. The p25 exposes this tail, which a median alone hides.
Per-pair gap matrix: for each ordered speaker pair (“B after A”), the median gap and sample count. This tells you who enters fast after whom; values require at least 3 samples to appear.

Question-flow classification

analyzeQuestionFlow() operates at the sentence level within utterances. Each question-marked sentence is first classified by dialogue act (DAMSL-lite, Stolcke et al. 2000):

question: interrogative lead (what, why, how, is, do, can, …).
check: statement-form confirmation (“So I use the admin SDK?”).
tag: declarative stem with a short question tail (“That’s with the NHID, right?”).
backchannel: phatic continuers in question form (“Really?”, “Is that right?”).

Tags and backchannels are excluded from the outcome analysis. For each question classified as question or check, the analyser scans forward up to a configurable window (default 10 seconds) for a response:

A different speaker’s substantive response (not a short backchannel) classifies as elaboration if it introduces new information, or assent if it matches an acknowledgement lexicon (“yeah”, “that makes sense”, “sounds good”, “fair enough” - 41 entries, matched greedily for multi-word phrases).
If only backchannels come from others and then the asker speaks next, the question is self-answered - the ask-then-self-answer move.
Nothing within the window: unanswered. The answer may arrive later; the label is low-confidence.

The classification is form-based and achieves about 70% accuracy against human-labelled dialogue-act corpora. The output carries a note to this effect, and the tool’s prompt guidelines tell the agent to verify flagged instances against the transcript before asserting them.

Two concrete limits follow from the form-based approach:

A steering question and an information-seeking question with the same surface form produce the same classification. The outcome (assent vs elaboration) carries a weak signal - when someone asks “did you try X?” and gets “yeah”, the asker was probably steering, but the heuristics report only the assent.
The check / question distinction depends on the interrogative-lead lexicon. “You tried the new endpoint?” is a check (no interrogative lead), while “Did you try the new endpoint?” is a question - but in conversation these are functionally the same act. The difference is counted but should not be over-interpreted.

Multi-unit turn blocks

Diarization fragments a held floor into many short turns because the model inserts segment boundaries on acoustic pauses. computeBlocks() reverses this: consecutive same-speaker utterances separated by no more than 3 seconds are merged into multi-unit-turn blocks. The result is a per-speaker block count, duration percentiles, and the share of total words spoken inside blocks lasting 30 seconds or more.

Long blocks are not inherently a defect. Expert answers, stacked questions, and detailed explanations are legitimately long. The top-five longest blocks are returned as evidence for the reviewer to read before judging. Verbatim phrase repetitions inside blocks (4+ tokens, 2+ content words, repeated within the same block) are flagged separately - these are the composing-aloud signature, not the topical repetition that an LLM would need to catch.

Longitudinal baselines

video_metrics writes a row per (speaker, bundle) to a JSON-lines file under the same cache directory. When the same speaker appears in later calls tagged with the same format, the tool prints a delta against the speaker’s prior results: change in fillers per 100 words, change in gap median, change in long-block share. The history is advisory only - a failed write does not block the metrics call. Re-running metrics on the same bundle replaces that bundle’s rows rather than duplicating them.

The frame-description endpoint and vision reuse

The /api/describe endpoint that video_extract uses for video frames is one of two vision endpoints the whisper stack exposes. The other is /api/image, which takes raw image bytes as a multipart upload and returns a description plus OCR text.

This second endpoint is used by vision-fallback.ts: when the default cloud model cannot read images (it is text-only), the extension replaces each image block in the outgoing request with a text description. The first-choice backend is the LLM engine if its loaded model has vision; the fallback is /api/image, which returns a self-contained description without needing a server-side file path. Descriptions are cached by image hash so each image is described once.

Both endpoints use the same VLM, which is reported in the service’s /api/status response alongside the GPU device and compute type.

What carries over to another setup

The model names, the GPU proxy’s port numbers and the MCP tool count are specific to this setup. These properties are not:

Return a summary, not the transcript. A raw diarized transcript is thousands of tokens. An agent’s context is scarce; filling it with every word of a recording that the model is not meant to memorise wastes its budget. Return speaker shares, a bundle path and a warning when something looks off; let the agent drill in with analysis tools that read from disk.
Cache the expensive step. Transcription and diarization are GPU-bound and slow. The analysis is pure math over the same data and instant. Write the extract result to disk once; every subsequent tool reads it without touching the GPU.
Word-level speaker timing is the foundation. Segment-level diarization rounds boundaries to utterance edges and hides real overlaps. Word-level timing with per-speaker utterance merging exposes acoustic collisions that segment timing cannot.
Voice prints are server-side, not per-client. Enrol once and every consumer of the same service gets named speakers: the web UI, the bot, the API, the agent. Embedding keys are stable diarized labels; display names are a separate layer that renames without breaking enrolment.
Heuristics are bounded; state the bound. Conversation metrics from form-based classification are about 70% accurate. Overlap cannot judge intent. A per-speaker gap p25 exposes the fast-tail that a median hides, but the classification of what that speed means stays a human call. State the limit and return the evidence so the reviewer can verify.
Long-running jobs use a queue and polling, not a blocking HTTP call. Transcription of a long recording outlasts any reasonable HTTP timeout. The agent tool polls a job-id endpoint; the model is not re-prompted while waiting.
One GPU, one workload at a time. Transcription evicts the LLM. The next chat turn pays a cold-context prefill penalty proportional to the session length. Size the model and the diarization decision against that cost: for a short recording you only need text from, skip diarization and pay a smaller eviction window.

Decision guide

The same thing as text:

Do you need per-speaker analysis (overlap, metrics, blocks)? If yes, run video_extract with diarize: true - this takes 8-10 minutes for a 60-minute recording. If you only need the transcript, set diarize: false (2-3 minutes).
Does the visual track matter (screen shares, slides)? Enable frames: true. Otherwise skip it.
Is the speaker already enrolled? If no, enrol with video_enroll from the cached bundle after extract, or relabel with video_name from a manual or LLM-inferred name map.
Run video_overlap for acoustic collision analysis. It returns who talked over whom and who yielded - objective, not interpretive.
Run video_metrics for speaking style: rate, fillers, turn-gap percentiles, question-flow classification. All are heuristic; verify top events against the transcript.
Run video_doc to assemble the evidence bundle, then let the agent write the review from it.

Evidence

Claim	How it was checked	Status
Default model is `turbo`; five other sizes available (`tiny` through `large`)	whisper skill SKILL.md lines 125-126 (`turbo` default, enum list)	Documented
Translation default is `"auto"` (30 s LID pre-pass, translate non-English)	whisper skill SKILL.md parameter table (`translate` default `"auto"`: “LID pre-pass, translate non-English to English”), whisper-transcribe docs/design/multilingual.md (”30s LID pre-pass via faster-whisper”)	Documented
`video_extract` polls `/api/jobs/<id>` every 4 seconds	`runTranscriptionJob()` in video-review-core.ts line ~1120 (`await sleep(4000)`)	Code-verified
Default timeout for transcription is 1800 seconds	`video_extract` parameter description in video-review.ts (“Max seconds to wait for transcription (default 1800).”)	Code-verified
Bundle cache path is `XDG_DATA_HOME/video-review/`	`CACHE_DIR` constant in video-review-core.ts line ~50 (`XDG_DATA_HOME ?? join(homedir(), ".local", "share"), "video-review"`)	Code-verified
Utterance merge gap is 0.6 seconds	`mergeUtterances()` default `gapSec = 0.6` in video-review-core.ts	Code-verified
Overlap minimum threshold is 0.3 seconds	`computeOverlap()` default `minOverlapSec = 0.3` in video-review-core.ts	Code-verified
Block merge tolerance is 3 seconds, long-block threshold is 30 seconds	`computeBlocks()` defaults `gapTolSec = 3, longBlockSec = 30` in video-review-core.ts	Code-verified
Question-response window is 10 seconds	`analyzeQuestionFlow()` default `windowSec = 10` in video-review-core.ts	Code-verified
DAMSL-lite form-based classification is ~70% accurate	Tool prompt guidelines + comment in `classifyQuestion()` (“expect machine-DA accuracy (~70%), not perfection”)	Documented
Voice-print embedding is 256-dimensional (WeSpeaker)	`Bundle` interface `speakerEmbeddings` field comment (“256-d WeSpeaker voice embeddings”) in video-review-core.ts	Code-verified
Whisper MCP server exposes 6 tools (whisper_status, yt_download, whisper_transcribe, yt_transcribe, yt_transcribe_playlist, wait_job) with `exposure: direct`	`mcp.json` entry for whisper (no `exposure` override = direct)	Code-verified
`/api/image` accepts raw multipart bytes and returns description + OCR	`vision-fallback.ts` lines 25-28 (“single-still VLM + OCR … accepts raw multipart bytes”)	Code-verified
Vision-fallback caches descriptions by image hash on disk	`vision-fallback.ts` lines 29-31 (“cached by image hash, in memory and on disk at `~/.cache/pi-vision-describe/<sha>.txt`”)	Code-verified
Whisper translate quirk: per-segment decoder can override translate on multilingual audio	`docs/design/multilingual.md` lines 26-30 (“faster-whisper’s per-segment decoder occasionally ignores the translate task token”)	Documented
Diarization triples extract time: ~2-3 min text-only vs 8-10 min with it on a 60-min call	LLM agent experience; not a measured benchmark	Observed
Eviction cold-prefill penalty: 40-90 s TTFT at ~200k context	`agent-model-routing.mdx` “Local-engine guards” table row for `llmc-session-guard.ts`	Documented

Scheduling one GPU between LLM, image and training work - the GPU proxy that the whisper stack shares
Which model runs which agent work - vision-fallback path and model routing rules
The pi harness journey - the series hub for the harness this stack runs inside
Loop or graph: a sensor-gated coding-agent loop - the loop that can gate on metrics from this pipeline

Linked from

The pi harness journey: a coding-agent harness built from one

Link graph: 1 doc links here, this page links to 4