Voice Architecture

Key components

• Retell.ai: session management, real-time ASR, and TTS. Provides interruption/barge-in signals, streams transcripts to Leena AI, and carries the telephony leg (including SIP transfer) for phone-based deployments.

• Leena AI Core (Gateway): owns the Retell session and everything around the call — call initiation and tokens, the bot kill switch, the one-active-call-per-user guard, transcript assembly into chat history, hold state, rich-content delivery (chat or SMS), phone-call authentication, and call analytics.

• Voice Orchestrator (perimeter fast path): a low-latency mini-agent running inside the Autonomous Agent service. It is deliberately small — temperature 0, a short output budget, and a four-tool surface. It:

  • Instantly acknowledges or clarifies
  • Provides micro-responses to small talk and confirmations
  • Gates and forwards complex turns to the core agent (converse_with_main_agent)
  • Controls the session: cancel a running task, end the call, or request a human transfer
  • Rewrites core-agent output into speech-friendly language (strips markdown, turns lists into spoken transitions)

• Core Orchestrator (Leena Autonomous Agent): planning, tool selection, execution, knowledge grounding, and long-running job coordination.

• Model routing: both orchestrators resolve their model through Leena's provider-agnostic LLM layer rather than a pinned model name — Azure OpenAI primary with OpenAI failover on the voice fast path — so models can be upgraded without a code change.

• State layer: live call state is held in memory on the pod owning the WebSocket, snapshotted to Redis (2-hour TTL) and broadcast over Redis Pub/Sub so any pod can recover a call; the final transcript is archived to MongoDB.

• Text-to-Speech (TTS): Retell.ai primary; voice persona per customer.

  

Request Lifecycle

• Call start: The client requests a call from Core, which checks the bot kill switch and rejects a second concurrent call for the same user. Core returns a call token; the client opens a secure WebSocket (WSS) to Retell.ai and audio streams bi-directionally.
• Greeting: The first turn (response_id 0) returns the configured greeting without a full model pass. The client keeps the microphone muted through the greeting to avoid speaker-to-mic echo, then unmutes automatically.
• Real-time ASR: Retell.ai emits partial/final transcripts to Leena over WSS (encrypted in transit).
• Perimeter fast-ack: The Voice Orchestrator consumes streaming text and issues an instant, sentiment-matched acknowledgement (e.g. "Got it — checking your PTO") without blocking on the core agent.
• Routing:
  • Simple/safe turns (greetings, confirmations, clarifications) are answered by the Voice Orchestrator directly
  • Transactional/complex turns go to the Core Orchestrator
• Execution (Core): intent grounding → plan → tool/API calls (e.g. HRIS). Context is kept minimal (see latency section) and cached when safe.
• Progress updates: The core agent streams machine-readable status ("checking KB", "calling Time-Off API"), which the Voice Orchestrator rephrases as natural speech. Filler updates escalate over the wait — short holds first, then reassurance, then an acknowledgement of complexity. If the core agent is blocked on the user (an OAuth sign-in, for example), the fillers switch to nudges instead of reassurance.
• Response delivery: Text leaves the orchestrator in chunks marked incomplete, and Core forwards each chunk to Retell.ai for synthesis as it arrives — so audio begins playing while the rest of the answer is still being written. Acknowledgements, progress updates, and final answers all travel this same streaming path.
• Rich content: Anything that cannot be spoken cleanly — links, tables, citations — is emitted as a separate chat message. On web calls it lands in the chat window alongside its sources; on phone calls it is delivered as an SMS. The chat copy is treated as canonical for that turn, so the spoken transcript is not persisted twice.
• Barge-in/interrupts:
  • Retell.ai signals user interrupt events mid-utterance and issues a new response_id
  • The orchestrator applies "latest turn wins": the in-flight stream is aborted and any running core-agent task is gracefully stopped before the new turn proceeds
  • Core reports back which portion of the previous answer was actually played, and the orchestrator tags the transcript accordingly — so it neither repeats what the caller already heard nor assumes they heard something that was cut off
• Mid-call actions: Hold and resume are client-initiated and broadcast to every open tab; hold state is persisted to Redis so it survives a pod restart, and time spent on hold is excluded from the chat transcript. Files uploaded from the chat composer are registered against the live call, acknowledged out loud on the open turn, and forwarded to the core agent with the current request.
• Human transfer (phone only): When the orchestrator requests a handover, Core validates the configured E.164 transfer number at dispatch time and Retell.ai performs the SIP transfer after the hand-off line is spoken. Web calls cannot be SIP-transferred.
• Phone-call authentication: Inbound callers are identified by phone number, then verified with either an OTP entered over DTMF or knowledge-based authentication (two mandatory identifiers plus a tiebreaker, matched against a hashed lookup), with per-step and per-flow retry limits.
• Observability: Per call, Leena stores metadata, audio, transcripts (chronological), and summaries for analytics and debugging, in the customer's hosting region. Call lifecycle events — initiated, connected (with failure reason), disconnected (with end type) — are emitted to the analytics pipeline.
• Resilience: Model and thread-dispatch failures retry with exponential backoff; if a WebSocket reconnects to a different pod, call state is rebuilt from memory → Redis → MongoDB.
• Auto Termination: A call ends automatically in three cases, and the agent always says why before the line drops:
  • No activity for ~20 seconds after the agent finishes speaking
  • The call exceeds the maximum duration (15 minutes)
  • A hold exceeds 5 minutes — with a spoken warning 30 seconds before disconnect

Latency design (why it feels instant)

We achieve near-instant acks (less than a human pause), smooth turn-taking, and fast task completion via:

• Streaming out: spoken text is emitted and synthesised chunk by chunk, so time-to-first-audio is set by the first few tokens rather than the full answer.
• Perimeter fast path: a small, fast model with a short output budget produces immediate, context-aware acks and simple replies.
• Streaming ASR: text tokens arrive as the user speaks; no end-of-speech blocking.
• Shallow turn path: the route from transcript to synthesised audio carries no redundant model passes between the orchestrator's decision and speech.
• Progressive disclosure: the user hears meaningful updates while the core agent executes.
• Context diet: the core agent strictly limits prompt/context size per turn; it uses selective retrieval and memoised tool results (caching) where permissible.
• Parallelism: plan and tool-prep run in parallel with TTS buffering of non-critical preambles; heavy events are dispatched to a task pool so the WebSocket never blocks on model latency, while heartbeats and transcript updates are handled inline.
• Vendor locality roadmap: future regional ASR/TTS to reduce RTT where needed.

Dialogue quality: prosody, sentiment, and control

• Naturalness default: modern TTS voices are humanlike even without SSML.
• Production control: for long or sensitive reads (spelling, reset steps), we apply SSML (pace, pauses, emphasis, repeat) on top of vendor defaults.
• Sentiment alignment: the Voice Orchestrator infers user sentiment and mirrors tone by selecting from acknowledgement families — neutral, upbeat, empathetic, supportive, appreciative — and by choosing phrasing and SSML cues to match.
• Voice-first rewriting: core-agent output is written for screens, so it is passed through a voice-specific rewrite before synthesis: markdown removed, lists converted into spoken transitions, length trimmed, and anything unspeakable diverted to chat or SMS.

Security, privacy, and data residency

• Transport security: All hops use WSS (TLS), bi-directional.
  • Client to Retell.ai (audio up)
  • Retell.ai to Leena (transcripts)
  • Leena to TTS to Retell.ai (audio down)
• Availability control: An admin-controlled kill switch disables an assistant instantly; incoming calls are rejected with a spoken message rather than failing silently.
• Retention controls at the subprocessor: Retell.ai is configured not to persist call data beyond a minimal operational window; Leena stores authoritative logs.
• Hosting region: Leena persists audio/transcripts in the same region as the customer's Leena tenant. Live call state lives in Redis with a 2-hour TTL and is evicted when the call ends.
• Constraint: Retell.ai currently hosts in US regions only; this can block strict-localisation customers (e.g. some ME countries, EU-only mandates).
• Roadmap: evaluate an in-house / open-source voice stack with pluggable ASR/TTS to unlock additional hosting regions and full data-plane control.