Vivoxx — On-Prem STT/TTS/LLM Voice Engine

Vivoxx — On-Prem STT/TTS/LLM Voice Engine

Second-generation LiveKit voice agent (sibling to FlowAgent TS) that adds a 3D talking-head avatar driven by visemes streamed inline from the on-prem Dudoxx CUDA Live TTS SSE service — pure on-prem STT/TTS/LLM pipeline with a strict LiteLLM-only LLM policy and per-tenant unified-settings cascade. Distilled from the canonical POC at ddx-livekit-mastra-rt-sandbox/ and flattened into production layout.

DRIFT NOTE (2026-07-02 re-verify): this shard's original W4 text described a Kokoro-TTS + HeadTTS + Deepgram-only + @livekit/agents-plugin-openai pipeline. The live code has moved on: TTS is a single dudoxx-tts provider (CUDA Live TTS, GET /v1/speak/sse) with visemes carried inline in the same SSE stream — the separate HeadTTS provider was deleted 2026-06-13 (T5 hard cutover). STT is two backends — CUDA Parakeet (dudoxx-stt, canonical) and licensed on-prem Deepgram — both deepgram-wire-shape. LLM routes through the LiteLLM gateway. The corrected facts below supersede the historical prose where they conflict.

Business Purpose

Voice without a face is good; voice with a clinician-shaped avatar that lip-syncs to PHI-safe TTS sells differently. Vivoxx is the productisation of that avatar story:

• No cloud STT/TTS/LLM — anywhere in the loop (LLM provider whitelist throws on anything other than LiteLLM).
• Visemes shipped inline from on-prem CUDA Live TTS — the avatar's mouth shape arrives in the same SSE stream as the audio; the browser never reconstructs phonemes from waveform.
• One Mastra-flavored agent surface — the Mastra Agent builder exists for tool calls today (Phase 1 ships factory-only) and becomes the runtime LLM later (Phase 2 — see code comment in vivoxx-agent.ts).

This is what we demo to investors who want to see "Dudoxx the brand" speaking with a credible clinician persona, and what we ship to clinics that want a virtual receptionist that does not depend on ElevenLabs / D-ID / HeyGen cloud rendering.

Audiences

• Investor: Lip-synced clinician avatar end-to-end on-prem. Differentiator vs Hippocratic AI demo (cloud TTS, cloud avatar).
• Doctor / clinical buyer: "On-prem voice + face" as a single SKU. Same Dudoxx voice + face across patient kiosks, reception, telemed waiting rooms.
• Developer/partner: Clean POC mapping table (reference_vivoxx_poc) — every Vivoxx file points to its POC origin so behaviour can be checked against the canonical reference.
• Internal (ops/support): Single Node process, room filter agent_name === 'vivoxx', secrets enforced (worker.ts:9 — "DO NOT log secrets"). TTS (with visemes) is the shared CUDA Live TTS service — no separate viseme service.

Architecture

ddx-vivoxx worker (tsx src/worker.ts via @livekit/agents CLI) — entered from a LiveKit room where metadata.mode === 'vivoxx' and agent_name === 'vivoxx'.

Phasing (current state vs planned — do not read Phase 2 as shipped):

Tech Stack & Choices

• Runtime: Node.js, @livekit/agents CLI. Distinct agent name vivoxx (vs ddx-flowagent-voice from livekit-flowagent-ts.md) — two agents coexist on one cluster, dispatched by room metadata.
• LLM: LiteLLM-only enforced at provider creation (providers/llm.provider.ts:38-44 — whitelist ['dudoxx','DUDOXX','LITELLM','litellm','DUDOXX_LITELLM','DUDOXX_LLM'], anything else throws). Default temperature 0.2, default model from config.llm.model.
• STT: stt.provider.ts selects between exactly two deepgram-wire backends — CUDA Parakeet (providers/stt/dudoxx-stt.ts, DudoxxSttSTT, stt.home.dudoxx.com, canonical) and licensed on-prem Deepgram (providers/stt/deepgram-onprem.ts, deepgram.home.dudoxx.com). Silero VAD owns turn detection. Any other provider value throws. Parakeet prepends a per-language audio primer (en/fr/de) from ?language= — pass meta.locale or non-EN garbles.
• TTS: a single dudoxx-tts provider (providers/tts.provider.ts → @ddx/voice-providers DudoxxTtsClient) streaming from CUDA Live TTS over GET /v1/speak/sse. No engine branching, no Kokoro. Any other provider value raises — no silent fallback.
• Visemes are inline in the TTS SSE (packages/ddx-voice-providers/src/clients/dudoxx-tts.client.ts): DudoxxTtsClient.stream() calls onVisemes(TtsViseme[]) synchronously before each PCM chunk (already Preston-Blair 15-set, no remap). The separate HeadTTS provider/client was deleted 2026-06-13 (T5 hard cutover); the browser never reconstructs phonemes. Measured TTFA ~290 ms (vs 1.27–11.6 s for the former blocking /v1/synthesize POST, now removed).
• Mastra agent (mastra/vivoxx-agent.ts:1-12): factory only in Phase 1. Each tool entry becomes a Mastra tool that POSTs to /voice/tools/execute with typed toolId. Phase 2 wires it as runtime LLM.
• Profile resolver (profiles/resolver.ts): agent field in room metadata + locale → resolved persona + voice + tool catalog.
• Unified settings cascade (clients/settings-client.ts): per-tenant STT/TTS/LLM overrides — matches the cascade in voice-processor.service.ts (see voice-commands.md).
• Tooling bridge: bridge/ddx-api.client.ts (DdxApiClient) is the only outward-facing tool surface. All tool execution funnels through POST /voice/tools/execute.
• SSE: bridge/sse-publisher.ts emits TranscriptChunk, ToolStartedEvent, ToolCompletedEvent, AgentStateChanged (see packages/ddx-sse-contract/, @ddx/sse-contract 4.1.0).
• Why a separate package from FlowAgent TS? Different Mastra runtime contract (Vivoxx uses Mastra Agent for tool wrapping, FlowAgent uses Mastra Workflow), different avatar surface (inline-viseme TTS stream), different LiveKit agent name. Sharing would muddy both.
• Why server-side visemes and not browser-side lipsync? Browser-side phoneme reconstruction (e.g. via WebAudio analyser) drifts under packet loss. The CUDA Live TTS SSE stream emits TtsViseme[] (Preston-Blair 15-set) synchronously ahead of each PCM chunk so the avatar engine renders mouth shapes in lock-step with audio.

Data Flow

1. Room created (ddx-api or ddx-web): metadata { mode: 'vivoxx', sessionId, userId, orgId, clinicId, locale, agent }.
2. Worker dispatched (worker.ts): parses metadata via VivoxxRoomMetadata type guard (worker.ts:48-65).
3. Profile resolved: resolveProfile(metadata.agent, locale) → resolved persona + voice + tool catalog.
4. Unified settings fetched: fetchUnifiedSettings(orgId, clinicId) — tenant-level STT/TTS/LLM overrides.
5. Providers created:
   • createStt(overrides) → DudoxxSttSTT (CUDA Parakeet) or DeepgramOnPremSTT.
   • createTts(overrides) → single dudoxx-tts provider (CUDA Live TTS SSE, inline visemes).
   • createLlm(overrides) → openaiPlugin.LLM({ baseURL: config.llm.baseUrl, … }) (providers/llm.provider.ts:36-55), LiteLLM gateway only.
6. Voice loop: LiveKit voice.AgentSession({ stt, tts, vad, llm }) runs the standard turn loop.
7. Tool call: when the LLM emits a tool call, DdxApiClient.toolExecute({ toolId, args, ctx }) POSTs to ddx-api /voice/tools/execute. Response → back to LLM as tool message.
8. Avatar (viseme path): each TTS request opens the CUDA Live TTS SSE (GET /v1/speak/sse via DudoxxTtsClient.stream()), yielding PCM frames plus TtsViseme[] surfaced synchronously through the onVisemes callback before each chunk → DataChannel → browser avatar engine. tts.provider.ts paces frames to LiveKit's ring buffer (VIVOXX_AUDIO_QUEUE_MS).
9. SSE: SsePublisher.publish('transcript' | 'tool_started' | 'tool_completed' | 'agent_state_changed', payload) → ddx-api SSE → ddx-web.
10. Session complete: DdxApiClient.completeSession(payload) finalises persistence.

Implicated Code

MANDATORY: ≥3 file:line citations.

• ddx-vivoxx/src/worker.ts:1-16 — entry point + secrets discipline (DO NOT log secrets).
• ddx-vivoxx/src/worker.ts:11-43 — imports: LiveKit agents, providers, bridge, profile resolver, settings client.
• ddx-vivoxx/src/worker.ts:48-65 — VivoxxRoomMetadata shape + locale guard.
• ddx-vivoxx/src/index.ts:1-35 — public runtime surface (DdxApiClient, SsePublisher, createStt/Tts/Llm, resolveProfile, buildVivoxxAgent).
• ddx-vivoxx/src/providers/llm.provider.ts:1-15 — preamble: "Direct OpenAI cloud is rejected — ALL LLM calls route through LiteLLM."
• ddx-vivoxx/src/providers/llm.provider.ts:36-62 — createLlm factory + whitelist enforcement + info return.
• ddx-vivoxx/src/providers/tts.provider.ts — single dudoxx-tts provider; delegates to @ddx/voice-providers DudoxxTtsClient, decodes 24 kHz PCM-S16LE, wall-clock pacing, onVisemes callback (HeadTTS provider deleted 2026-06-13).
• packages/ddx-voice-providers/src/clients/dudoxx-tts.client.ts — DudoxxTtsClient.stream(): GET /v1/speak/sse, parseSseStream, per-Audio-frame PCM + inline TtsViseme[]; blocking /v1/synthesize removed.
• ddx-vivoxx/src/mastra/vivoxx-agent.ts:1-15 — Mastra Agent factory (Phase 1 — not yet in voice loop).
• ddx-vivoxx/src/mastra/vivoxx-agent.ts:25-35 — toolInputSchema / toolOutputSchema (Zod) for /voice/tools/execute proxy.
• ddx-vivoxx/src/bridge/ddx-api.client.ts — DdxApiClient (tool execution + session completion).
• ddx-vivoxx/src/bridge/sse-publisher.ts — typed SSE publisher (TranscriptChunk, ToolStartedEvent, ToolCompletedEvent, AgentStateChanged).

Operational Notes

• POC reference (CRITICAL): ~/projects/sandboxes/dudoxx_clinic_production_sandbox/ddx-livekit-mastra-rt-sandbox/ — before implementing ANY new vivoxx feature, read the POC equivalent (memory shard reference_vivoxx_poc). POC is canonical for viseme streaming pattern, avatar registry, TTS sanitizer, tool bridge, guardrails / handoff / tool-approval (latter three not yet ported).
• POC layout mapping (from reference_vivoxx_poc):
  • ddx-livekit-agent/src/agent.ts → ddx-vivoxx/src/index.ts + worker.ts
  • ddx-livekit-agent/src/core/create-ddx-agent.ts → ddx-vivoxx/src/mastra/vivoxx-agent.ts
  • ddx-livekit-agent/src/providers/avatar/headtts-avatar.ts → superseded; visemes now inline in ddx-vivoxx/src/providers/tts.provider.ts (HeadTTS provider deleted 2026-06-13)
  • ddx-livekit-agent/src/providers/tts/sanitizer.ts → missing in vivoxx; port if speech artifacts appear
  • ddx-livekit-agent/src/core/guardrail-pipeline.ts → not yet in vivoxx
  • ddx-livekit-agent/src/core/handoff-manager.ts → not yet in vivoxx
  • ddx-livekit-agent/src/core/tool-approval-manager.ts → not yet in vivoxx
• HeadTTS is GONE: the standalone HeadTTS viseme service (formerly ddx-vivoxx-headtts :6881) was deleted 2026-06-13. Visemes now ship inline in the CUDA Live TTS SSE stream — there is no separate viseme service to health-check.
• Secrets: LIVEKIT_API_SECRET, DDX_API_GATEWAY_KEY, LITELLM_API_KEY — NEVER log (worker.ts:9-10).
• Provider whitelist must be kept in sync with LiteLLM config — adding a new provider here requires the gateway alias.
• LiveKit key drift pitfall: same as FlowAgent — API key + secret MUST match across ddx-api + ddx-vivoxx + ddx-mastra-codebase. Mismatch presents as silent "Connecting…" hang.
• Mastra phase status: factory only today (Phase 1). DO NOT wire the Mastra Agent into the voice loop without reading the Phase 2 comment in vivoxx-agent.ts first.
• Viseme pacing pitfall: queueing the whole PCM blob at once produces "chunky/garbled" audio in the browser — LiveKit's audio buffer underflows. 20 ms framing with wall-clock pacing is non-negotiable.
• POC sanitizer port candidate: if you see speech artifacts (emoji, markdown leaking into synth), port providers/tts/sanitizer.ts from the POC.

Related Topics

• LiveKit FlowAgent TS Bridge — sister LiveKit Agent runtime; same patterns, different workflow
• Voxial LLM — same LiteLLM-only policy enforced here
• Live STT — DEPRECATED Whisper service; Vivoxx uses CUDA Parakeet / on-prem Deepgram
• Live TTS — DEPRECATED Kokoro service; Vivoxx uses the single dudoxx-tts CUDA Live TTS provider (inline visemes)
• LiveKit Infra (W1) — cluster + auth + URL
• FlowAgent Scenarios — workflow-driven sibling product