On-Prem STT/TTS/LLM — Self-Hosted Voice and Inference Stack

On-Prem STT/TTS/LLM — Self-Hosted Voice and Inference Stack

Self-hosted STT + TTS + LLM-router inference services unified behind LiteLLM and the production CUDA embedder — every PHI-touching token of speech and text stays on Dudoxx infrastructure.

DRIFT NOTE (2026-07-02 re-verify): the original W4 text described the Kokoro TTS stack (ddx-live-tts, kokoro-tts.plugin.ts, DEFAULT_TTS_PROVIDER=DUDOXX_KOKORO, af_heart voice) and the Whisper/FunASR STT stack (ddx-live-stt). Both of those FastAPI services are now DEPRECATED and moved to dudoxx-omni-intelligence (Kokoro fully retired 2026-06-12; see live-stt.md / live-tts.md). The live ddx-api voice path is:

• TTS → provider DUDOXX_TTS (CUDA Live TTS, Qwen3+XTTS), default voice ddx_heart, plugin DudoxxTTSPlugin in livekit/plugins/dudoxx-tts.plugin.ts, env VOICE_TTS_DUDOXX_TTS_URL=https://tts.home.dudoxx.com (NOT KOKORO_TTS_*).
• STT → engine enums DUDOXX_DEEPGRAM (default) / DUDOXX_WHISPER / DUDOXX_STT (Forge multi-engine gateway); the canonical on-prem voice-agent STT is CUDA Parakeet via DUDOXX_STT.

Historical Kokoro/Whisper/FunASR prose below is retained for lineage; where it conflicts with this note, this note wins.

Business Purpose

The on-prem voice-and-inference stack is the single most expensive and the single most defensible piece of the Dudoxx platform. It exists because no SaaS combination of Deepgram + ElevenLabs + OpenAI is legally usable for a German clinic's dictation pipeline without a DPA chain that takes months to negotiate, and even then PHI leaves the country.

By owning the four hot-path inference services — STT (ddx-live-stt), TTS (ddx-live-tts), LLM (dudoxx-gemma via LiteLLM), embeddings (dudoxx-embed CUDA) — Dudoxx gains:

• HIPAA / GDPR / BfArM data residency by construction — voice → transcript → LLM → embedding can all complete without leaving the cluster. Cloud providers are optional add-ons, not the default.
• Zero per-minute STT/TTS cost — clinics pay for hardware once, then dictate unlimited hours. Deepgram at ~$0.0043/min × 200 dictation min/doctor/day × 30 doctors quickly outruns a single 4090 GPU.
• Latency floor under 200 ms — STT runs in the same datacenter / on the same LAN; round-trip to AWS US-East from Hamburg adds ~120 ms before any inference happens.
• Provider-portable contracts everywhere — TTS calls go to the Kokoro/Qwen3 endpoint via infra-litellm, LLM calls go through the same proxy, and embedding calls share the production CUDA endpoint at https://embedder.home.dudoxx.com/v1/embeddings. Swapping a model is a config change, not a code change.

Revenue lever: the "on-prem AI bundle" is the upsell that makes Dudoxx defensible against bundled GP suites. Every competitor that ships AI by re-selling OpenAI/Deepgram is exposed to data-residency objections; Dudoxx is not.

Audiences

• Investor: ownership of STT + TTS + LLM + embedder is the unfair-advantage tier of the moat — and the answer to "how does your margin survive at scale?". A clinic with 30 dictating doctors pays cloud STT/TTS what it costs to amortise one self-hosted GPU per year.
• Clinical buyer (doctor/nurse/receptionist): invisible — they hear "DDX scribe" / "voice command" / "AI assistant". The HIPAA/data-residency argument lives in procurement, not at the desk.
• Developer/partner: each service is a FastAPI process with a public WebSocket and REST surface; all server-side consumers in ddx-api/src/voice-video/ import a thin plugin (KokoroTTSPlugin, voice processor) and route through the standard config resolution chain (frontend > user-db > org-db > env defaults).
• Internal (ops/support): four endpoints to keep alive — localhost:6350 (Whisper STT), localhost:6360 (FunASR STT), localhost:8890 (Kokoro TTS via Docker — alias kokoro.dudoxx.com), localhost:4000 (LiteLLM gateway). A clinic where dictation suddenly fails almost always has one of these four health checks red.

Architecture

ddx-api (NestJS, port 6100) components

Boundaries:

• STT split (Whisper vs FunASR) — Latin scripts (EN/DE/FR/ES/IT/PT) → Whisper (ddx-live-stt :6350, faster-whisper on CUDA or MLX on Apple Silicon). CJK + auto-detect short utterances → FunASR (ddx-live-stt :6360, SenseVoiceSmall). Both speak the same /v1/listen WebSocket protocol; the engine choice is org-level (whisperSttEndpoint vs funasrEndpoint columns on organizations).
• TTS unification — Kokoro-82M (54 voices, 9 langs), Qwen3-TTS-0.6B (11 voices, EN/FR/DE), Piper TTS (DE), Chatterbox (emotion). All exposed by ddx-live-tts on :8890 behind POST /tts/stream; engine routing is automatic based on voice-id prefix (*_q → Qwen3, German voices → Piper, etc.).
• LLM unification — every chat-completion request goes through LiteLLM gateway :4000 regardless of upstream (self-hosted dudoxx-gemma, OpenAI, Anthropic). The USE_DUDOXX_LITELLM=true flag makes the on-prem path the global default.
• Embedder is shared — same https://embedder.home.dudoxx.com/v1 CUDA endpoint that powers the RAG sidecar (see rag-system-indexation). One model (dudoxx-embed, dim 2560) across every embedding consumer.

For LiveKit voice-agent orchestration on top of this stack see infra-livekit and voice-commands.

Tech Stack & Choices

• ddx-live-stt (FastAPI + faster-whisper / mlx-whisper / FunASR) — Python 3.11+, WebSocket-first, hot-reload, Prometheus metrics, API-key auth. Backends auto-selected: CUDA → faster-whisper (float16), Apple Silicon → mlx-whisper (int8), CPU → faster-whisper (int8). Two engine policies: LocalAgreement-2 (stable) or SimulStreaming/AlignAtt (ultra-low latency).
• ddx-live-tts (FastAPI multi-engine) — Kokoro-82M ONNX + Qwen3-TTS 0.6B + Piper + Chatterbox Multilingual. Docker target kokoro-tts-cpu or kokoro-tts-gpu. Streams over WebSocket and chunked HTTP for LiveKit. 24 kHz native Qwen3 quality; click-suppression already applied at v6.3.0.
• LiteLLM as the one model entrypoint — explicit choice over LangChain / vllm / custom proxy because it gives us: spend ledger, virtual keys, OpenAI-SDK compatibility, model aliasing. Detail in infra-litellm.
• dudoxx-gemma as the default model — set globally by USE_DUDOXX_LITELLM=true + DUDOXX_BASE_URL=https://llm-router.dudoxx.com/v1. The default thinking-off invariant (enable_thinking:false) is enforced cluster-wide for cost predictability; verified in the RAG sidecar default LLM contract.
• Production CUDA embedder external — embedder.home.dudoxx.com/v1/embeddings, dudoxx-embed, dim 2560, ~70 ms typical. Same endpoint everywhere — no duplicate embedding spaces.
• Why NOT all-cloud (Deepgram / ElevenLabs / OpenAI): PHI residency, latency floor, per-minute cost, and vendor lock are each individually disqualifying — together they make the cloud path commercially impossible for a German clinic at scale.
• Why NOT all-on-cluster GPU: small clinics without a GPU still need to deploy. The architecture supports a "thin clinic" tier that reaches out to a regional Dudoxx-managed GPU node over Traefik; the local clinic still runs the FastAPI proxies, just configures them to point at a shared upstream.

Data Flow

Dictation / live transcription (Whisper path)

1. Browser / LiveKit voice agent opens WebSocket to ws://localhost:6350/v1/listen (or org-configured endpoint).
2. ddx-live-stt streams audio frames into faster-whisper or MLX-whisper; emits partial + final segments with confidence + timing.
3. voice-processor.controller.ts posts the final transcript to VoiceProcessorService.processTranscription().
4. Two LLM agents run in Promise.all:
   • Correction agent — fixes ASR errors, emits structured corrections.
   • Intent agent — classifies intent (SUBMIT | CANCEL | CLEAR | NEW_LINE | NAVIGATE | …) plus a stripText (transcript with the intent verb removed).
5. Both agents speak to LiteLLM via the OpenAI SDK; LiteLLM resolves dudoxx-gemma and forwards to DUDOXX_GEMMA_API_BASE.
6. Response returned synchronously; SSE side-channel optional for streaming partials.

LiveKit voice-agent turn (Kokoro TTS path)

1. LiveKit voice agent (Python or TS) decides to speak; calls TTSService.synthesize({ input, voice, language }).
2. TTSService resolves endpoint/apiKey via VoiceConfigService (user > org > env defaults) and delegates to KokoroTTSPlugin.
3. Plugin POSTs to https://kokoro.dudoxx.com/tts/stream (or http://localhost:8890/tts/stream in dev) with bearer KOKORO_TTS_TOKEN.
4. ddx-live-tts routes to the engine matching the voice-id prefix (_q → Qwen3, de_* → Chatterbox, else Kokoro) and streams audio chunks back.
5. LiveKit agent pipes audio into the room track. No bytes touch ddx-web — TTS is server-to-server only.

TUCAN play_audio_feedback (UI tool path)

1. TUCAN agent inside ddx-api decides to speak to the user (e.g. "Saved.") and calls the play-audio-feedback tool.
2. The tool synthesises via Kokoro (default voice af_heart, English) and ships the audio over the SSE ui:action.play_audio_feedback event.
3. Frontend useTUCANUIActions plays the buffer — see ddx-ai-ui-events-frontend skill.

LLM completion (any agent or service)

1. Any NestJS service builds an OpenAI-SDK chat-completion with model: 'dudoxx-gemma'.
2. SDK posts to ${LITELLM_BASE_URL}/v1/chat/completions with Authorization: Bearer ${LITELLM_MASTER_KEY} (or per-org virtual key).
3. LiteLLM enforces budget, records spend, forwards to DUDOXX_GEMMA_API_BASE.
4. enable_thinking:false is injected by the RAG sidecar and is the cluster-wide default — no reasoning_tokens are billed.

Implicated Code

MANDATORY: ≥3 file:line citations.

• ddx-live-stt/README.md:135 — STT_PORT=6350 (default Whisper port).
• ddx-live-stt/README.md:144 — BACKEND=auto # 'auto', 'faster-whisper', 'mlx-whisper' (device-aware backend selection).
• ddx-live-stt/README.md:263 — backend matrix: GPU → faster-whisper float16, Apple Silicon → mlx-whisper int8, CPU → faster-whisper int8.
• ddx-api/.env.example:328 — STT_ENDPOINT=http://localhost:6350 (default Whisper HTTP endpoint).
• ddx-api/.env.example:329 — STT_WS_URL=ws://localhost:6350/v1/listen (default Whisper WebSocket).
• ddx-api/.env.example:335 — FUNASR_ENDPOINT=http://localhost:6360 (default FunASR endpoint — CJK + auto language).
• ddx-api/.env.example:344 — # TTS Provider: DUDOXX_KOKORO | ELEVENLABS | OPENAI.
• ddx-api/.env.example:345 — DEFAULT_TTS_PROVIDER=DUDOXX_KOKORO.
• ddx-api/.env.example:352 — KOKORO_TTS_ENDPOINT=https://kokoro.dudoxx.com (production TTS upstream).
• ddx-api/.env.example:355 — # KOKORO_TTS_ENDPOINT=http://localhost:8890 (dev override).
• ddx-api/prisma-main/models/enums.prisma:203 — WHISPER_LIVEKIT // Port 6350, MLX-Whisper/Faster-Whisper, ~200ms latency (engine enum).
• ddx-api/prisma-main/models/organization.prisma:58 — whisperSttEndpoint String? @default("http://localhost:6350") (per-org Whisper override).
• ddx-api/prisma-main/models/organization.prisma:63 — whisperSttWsEndpoint String? @default("ws://localhost:6350/v1/listen") (per-org WS override).
• ddx-api/prisma-main/migrations/20260211000000_add_stt_engine_settings/migration.sql:15 — historical migration moved orgs off legacy port 4300 onto 6350.
• ddx-api/src/platform/common/config/settings.config.ts:303 — DEFAULT_WHISPER_ENDPOINT = process.env.STT_ENDPOINT || 'http://localhost:6350'.
• ddx-api/src/platform/common/config/settings.config.ts:311 — DEFAULT_STT_ENGINE: SttEngine = (process.env.DEFAULT_STT_ENGINE as SttEngine) || 'WHISPER_LIVEKIT'.
• ddx-api/src/platform/common/config/settings.config.ts:316 — DEFAULT_FUNASR_ENDPOINT = process.env.FUNASR_ENDPOINT || 'http://localhost:6360'.
• ddx-api/src/platform/common/config/settings.config.ts:319 — DEFAULT_FUNASR_MODEL = process.env.FUNASR_MODEL || 'iic/SenseVoiceSmall'.
• ddx-api/src/platform/common/config/settings.config.ts:327 — DEFAULT_TTS_PROVIDER: TTSProvider = (process.env.DEFAULT_TTS_PROVIDER as TTSProvider) || 'DUDOXX_KOKORO'.
• ddx-api/src/platform/common/config/settings.config.ts:328 — DEFAULT_TTS_VOICE = process.env.DEFAULT_TTS_VOICE || 'af_heart'.
• ddx-api/src/platform/common/config/ddxllm.config.ts:48 — USE_DUDOXX_LITELLM = process.env.USE_DUDOXX_LITELLM === 'true' (global on-prem flag).
• ddx-api/src/platform/common/config/ddxllm.config.ts:49 — DUDOXX_BASE_URL ||= 'https://llm-router.dudoxx.com/v1' (default LLM router).
• ddx-api/src/voice-video/tts/tts.service.ts:1 — module header — Proxies TTS requests to the Kokoro TTS server (ddx-live-tts).
• ddx-api/src/voice-video/tts/tts.service.ts:70 — ISO_TO_KOKORO language-code map (ISO 639-1 → Kokoro single-letter codes).
• ddx-api/src/voice-video/voice/voice-processor.service.ts:1 — VoiceProcessorService class doc — parallel correction + intent agents via Promise.all.
• ddx-api/src/voice-video/livekit/plugins/kokoro-tts.plugin.ts:98 — export class KokoroTTSPlugin — LiveKit voice-agent TTS plugin.
• ddx-api/src/ai/tucan/tools/ui/play-audio-feedback.tool.ts:71 — Supports ddx-live-tts multi-engine: Kokoro (9 langs, 54 voices), Qwen3 (11 voices), Chatterbox (emotion), Fish Speech.
• ddx-api/src/ai/tucan/tools/ui/play-audio-feedback.tool.ts:104 — .enum(['auto', 'kokoro', 'chatterbox', 'fish']) — engine selector.
• ddx-docker-litellm/config/litellm-config.yaml:17 — dudoxx-gemma model definition + api_base: os.environ/DUDOXX_GEMMA_API_BASE.
• ddx-live-tts/docker-compose.yml:13 — PORT=8890 (Kokoro TTS server port).
• ddx-live-tts/README.md (header) — engine list — Kokoro-82M + Qwen3-TTS 0.6B + Piper TTS + Chatterbox Multilingual.

Operational Notes

• Four health checks own the stack (run from any clinic host):
  bash copy

  curl -sf http://localhost:6350/health           # Whisper STT
  curl -sf http://localhost:6360/health           # FunASR STT
  curl -sf http://localhost:8890/health           # Kokoro TTS
  curl -sf http://localhost:4000/health           # LiteLLM gateway
  

  Red on any of these → dictation / voice agent / TUCAN audio fail. Most "AI is broken" tickets resolve to one of these four.
• STT engine choice is per-org, not per-user — switching a clinic from Whisper to FunASR is an organizations row update on whisperSttEndpoint / funasrEndpoint. The web UI exposes this in settings-stt-engine (see ddx-web specialist memory).
• Kokoro voice ID convention: af_heart, af_bella, bm_george (Kokoro), *_q suffix → Qwen3 (e.g. af_heart_q, df_katharina), German thorst* → Piper. The frontend play_audio_feedback tool validates against this list (play-audio-feedback.tool.ts:39).
• KOKORO_TTS_ENDPOINT is the legacy env name — prefer KOKORO_TTS_URL. The client SDK in ddx-live-tts/clients/ts/ warns once on legacy use. New deployments MUST use the new name.
• HIPAA boundary inside LiteLLM — any prompt containing FHIR PHI MUST set model: 'dudoxx-gemma'. LiteLLM does NOT inspect content. Code review is the line of defense — see infra-litellm "HIPAA boundary".
• Embedder is a hard dependency — both the RAG sidecar AND in-product semantic search hit embedder.home.dudoxx.com. An air-gapped clinic must run a local mirror; production CUDA endpoint is the default at ~70 ms.
• enable_thinking:false is injected for dudoxx-gemma everywhere — verified in ~/.claude/CLAUDE.md (M-USER 2026-05-17). Verify by checking reasoning_tokens: 0 in the metrics block returned by rag_investigate.
• GPU sharing: the Whisper service can run on CPU (int8) for receptionist call quality but NOT for dictation. Real dictation needs CUDA float16 or Apple-Silicon MLX int8. See backend matrix at ddx-live-stt/README.md:263.
• Recording archive: dictation audio is persisted to MinIO bucket {clinic-slug}-recordings — see infra-minio and infra-jitsi (shares the recording webhook pattern).

Related Topics

• infra-litellm — the unification point for every LLM call; sets the model alias, budget, virtual-key story
• infra-livekit — voice-agent RTC; the STT/TTS pipeline plugs in via KokoroTTSPlugin + Whisper WS
• infra-jitsi — sibling video stack; NOT a substitute for the voice pipeline
• live-stt — deeper dive into the Whisper / FunASR engine details
• live-tts — deeper dive into Kokoro / Qwen3 / Piper / Chatterbox engine selection
• voxial-llm — voice-LLM agent integration layer (Mastra-side)
• voice-commands — frontend command UX powered by VoiceProcessorService
• rag-system-indexation — shares the production CUDA embedder embedder.home.dudoxx.com
• CLAUDE_DEPLOYMENT_STT_ENGINE — companion deployment guide