/Knowledge Base
GitHub
07-voice-and-telemedwave: W4filled18 citations

Live STT — Real-Time Speech-to-Text

Audiences: developer, internal

Live STT — Real-Time Speech-to-Text

⚠️ DEPRECATED (2026-06). The ddx-live-stt FastAPI service documented here has been moved out of this monorepo to dudoxx-omni-intelligence. It survives in-tree only until the client migration completes (see root CLAUDE.md "Deferred (NOT archived, next-session wipe)" + ddx-reminders/). The live on-prem STT for voice agents is now the CUDA Parakeet backend (NeMo Parakeet-TDT, stt.home.dudoxx.com / stt.forge.dudoxx.com), consumed by Vivoxx via ddx-vivoxx/src/providers/stt/dudoxx-stt.ts and by ddx-api via the DUDOXX_STT engine. Do NOT wire NEW consumers to ddx-live-stt. Historical description below.

Self-hosted, on-prem real-time transcription service that streams clinical-grade Whisper output to TUCAN voice agents, FlowAgent scenarios, and Vivoxx — so every utterance the patient or clinician makes hits a Mastra agent in <500 ms with no audio leaving Dudoxx infrastructure.

Business Purpose

Voice-driven workflows (TUCAN Operator, FlowAgent voice forms, Vivoxx avatars) all need low-latency transcription that is HIPAA-compliant by construction — meaning the audio stream MUST NOT leave the customer's VPC. ddx-live-stt is the on-prem speech recognition tier that replaces Deepgram Cloud for tenants under data-residency mandates and provides a deterministic fallback when the cloud STT path degrades. It is licensed as part of the Dudoxx Voice Pack and is the first piece of the "no PHI to OpenAI / no audio to Google" story we sell to hospital procurement.

Audiences

  • Investor: Removes a recurring per-minute STT bill (Deepgram cloud) and unblocks deals where the customer refuses to send audio to US-hosted SaaS. Margin lever + sales unblocker.
  • Clinical buyer (doctor/nurse/receptionist): Voice assistant works during internet outages and in EU/CH/DE deployments where cross-border audio is legally constrained.
  • Developer/partner: Single FastAPI service exposing WS /ws/transcribe (DDX STT Protocol v2). Same WebSocket protocol whether the backend is faster-whisper on CUDA, MLX on Apple Silicon, or a future engine plug-in.
  • Internal (ops/support): Pre-loads a default model at boot, has /health + /metrics routes, and degrades gracefully (model fails to load → service stays up, requests get a structured error).

Architecture

Rendering diagram…

AudioSession + Pipeline stages

StageDetail
VadStageSilero
AsrStageDdxTranscriptionEngine — LocalAgreement-2 strategy, backends: FasterWhisperBackend (CUDA / CPU), MlxWhisperBackend (Apple Silicon)
DiarizationStagesortformer | pyannote
PunctuationStage—

Layered NLAH split: transport (api/) is isolated from inference (engine/); the pipeline (pipeline/) chains stages with shared PipelineContext. Vendored Whisper code lives under engine/vendored/ so the service has zero git-runtime dependency on upstream openai-whisper — critical for air-gapped customer deployments.

Tech Stack & Choices

  • Framework: FastAPI + Uvicorn (ddx-live-stt/pyproject.toml:14-30) — async WebSocket support, Pydantic v2 schemas, lifespan hooks for model pre-load.
  • Models: large-v3-turbo by default (ddx-live-stt/src/ddx_live_stt/settings.py:50-53), with the full Whisper catalog (tiny → large-v3) selectable per session.
  • Inference backends: faster-whisper (CTranslate2 — CUDA + CPU int8) and mlx-whisper (Apple Silicon native). Auto-detected (ddx-live-stt/src/ddx_live_stt/settings.py:54-65).
  • Streaming strategy: LocalAgreement-2 (engine/ddx_engine.py:43-50) — two consecutive Whisper passes must agree before a segment is finalised. Keeps the inference buffer ~2-5 s instead of re-transcribing every 15 s. Alternative simul-streaming (AlignAtt) selectable via engine_policy.
  • VAD: Silero ONNX (audio/vad_model.py) — short-circuits silent frames before they hit Whisper.
  • Diarization: NVIDIA Sortformer (default) or pyannote / diart (schemas/config.py + pipeline/diarization.py).
  • Why not call OpenAI / Deepgram Cloud directly? Audio is PHI. The whole reason this service exists is that the cloud route is legally unusable in our EU/CH targets.

Data Flow

  1. Client opens WS /ws/transcribe?language=en&model=large-v3-turbo and authenticates with DDX_LIVE_STT_API_KEY (header or query param — see api/auth.py).
  2. Server responds with SessionCreated (schemas/ws.py) — includes session_id, model name, sample rate.
  3. Client streams 16 kHz mono PCM as binary WS frames; control messages ({"type":"flush"}, {"type":"stop"}, {"type":"config", ...}) come as text frames.
  4. A receiver task enqueues frames; a separate processor loop drains them, runs VAD → ASR → diarization → punctuation, and emits:
    • VadSpeechStart / VadSpeechEnd
    • TranscriptionInterim (each pass)
    • TranscriptionFinal (LocalAgreement-2 confirmed; with optional words: WordTimestamp[])
    • ErrorMessage on engine fault
    • SessionEnded on close
  5. Consumers — ddx-livekit-flowagent-ts/src/index.ts for FlowAgent voice, ddx-api/src/voice-video/livekit/services/pipeline/voice-stream.service.ts for TUCAN Operator — read TranscriptionFinal.text and forward it to the Mastra agent / workflow step. No SSE involvement: the WS is point-to-point between the consumer and STT.

Implicated Code

MANDATORY: ≥3 file:line citations.

  • ddx-live-stt/src/ddx_live_stt/app.py:25 — FastAPI app factory; lifespan pre-loads default engine (app.py:54-67).
  • ddx-live-stt/src/ddx_live_stt/app.py:133-141 — router mount: test page, health, config, models, metrics, ws_router (WebSocket), admin.
  • ddx-live-stt/src/ddx_live_stt/api/ws_transcribe.py:1-44 — DDX Live STT Protocol v2 WebSocket implementation; producer-consumer split keeps the event loop responsive while MLX inference blocks.
  • ddx-live-stt/src/ddx_live_stt/engine/ddx_engine.py:43-50 — DdxTranscriptionEngine (LocalAgreement-2 online Whisper).
  • ddx-live-stt/src/ddx_live_stt/engine/backends/faster_whisper_backend.py:1 — CTranslate2-backed engine (CUDA / CPU int8).
  • ddx-live-stt/src/ddx_live_stt/engine/backends/mlx_backend.py:1 — Apple Silicon MLX backend (vendored simul_whisper).
  • ddx-live-stt/src/ddx_live_stt/settings.py:35 — default port 6350.
  • ddx-live-stt/src/ddx_live_stt/settings.py:46-49 — engine_policy switch (local-agreement-2 | simul-streaming).
  • ddx-live-stt/src/ddx_live_stt/settings.py:50-65 — model + backend + device + compute selection.

Operational Notes

  • Port: 6350 (settings.py:35). NOT 6360 — there is no in-tree FunASR engine today; the W4 brief that referenced it was speculative. If a FunASR backend is added later it would slot in under engine/backends/.
  • Model pre-load: lifespan loads the default engine at startup and logs load time (app.py:58-67). A load failure does NOT kill the service — it serves errors per session until a working model is reachable.
  • Env vars: STT_PORT, MODEL_NAME, BACKEND (auto/mlx-whisper/faster-whisper), DEVICE (auto/cuda/mps/cpu), COMPUTE_TYPE, API_KEY, DEFAULT_LANGUAGE. See ddx-live-stt/src/ddx_live_stt/settings.py:23-79.
  • Auth: empty api_key disables auth (dev mode); production deployments MUST set it (settings.py:42).
  • Pitfall — backend mismatch on workstations: developer Macs auto-pick MLX; CI/CD on Linux picks faster-whisper. Word-timestamp formats differ slightly; always test through the WS contract, not against the backend directly.
  • Pitfall — buffer trimming: buffer_trimming_sec default 15 s (settings.py:77-79). Lower for chatty workflows; raising it improves accuracy on long monologues but linearly increases re-transcription cost.
  • Defects: any STT-side regressions go into DEFECTS/ with area: voice/stt and a copy of the failing TranscriptionFinal sequence.
  • REMINDER: STT is the only voice service that touches raw audio. Audit access carefully — api_key is the only thing standing between an attacker and the patient's microphone.

Related Topics