Voice Architecture

Leena AI runs voice calls on two versions while voice 1.0 is being retired. Voice 2.0 uses a native speech-to-speech model; voice 1.0 uses the Retell-based pipeline of speech recognition, a language model and speech synthesis. On both, every request is handled by the same Leena AI orchestrator as chat. This page describes each version in full.

At a glance

Voice 2.0

Architecture overview

Call audio travels directly between the employee and the speech-to-speech model — over WebRTC from the web app, or over SIP from a phone. Leena AI never carries the audio. It runs everything around the call: who is calling, what the model is allowed to do, and the work behind every request.

flowchart TB
    subgraph CALLERS["Employees"]
        direction LR
        WEB["Web app<br/>(browser)"]
        PH["Phone caller"]
    end

    subgraph MEDIA["Call audio"]
        direction LR
        RT["OpenAI GPT Realtime<br/>speech-to-speech model"]
        TW["Twilio number<br/>+ SIP trunk"]
    end

    subgraph LEENA["Leena AI platform (your hosting region)"]
        direction LR
        CORE["Leena AI Core<br/>(Gateway)"]
        VR["Voice runtime"]
        MO["Master orchestrator"]
        TOOLS["Tools, AOPs, knowledge,<br/>connected systems"]
        STORE[("Redis + MongoDB<br/>call state, transcripts")]
    end

    WEB == "audio (WebRTC)" ==> RT
    PH == "audio" ==> TW
    TW == "SIP" ==> RT

    WEB -- "start call, cards in chat" --- CORE
    RT -- "incoming-call webhook (phone)" --> CORE
    RT -- "sideband: transcripts, tool calls, speech" --- VR
    CORE -- "session, events" --- VR
    VR -- "requests" --> MO
    MO --> TOOLS
    CORE --- STORE

Key components

• OpenAI GPT Realtime (speech-to-speech model): hears the caller and speaks back in one model, with no separate transcription and synthesis steps. It detects when the caller has finished speaking, stops when interrupted, and transcribes the caller's speech for the chat record. The default model is gpt-realtime-2.1, with gpt-4o-transcribe for caller transcripts.
• Leena AI Core (Gateway): owns everything around the call — call initiation and per-call credentials, the one-active-call-per-user guard, admission of incoming phone calls, caller identification (OTP or KBA), the bot kill switch and spam limits on phone, transcript assembly into chat history, hold state, rich-content delivery (chat or SMS), call history, and call analytics.
• Voice runtime: a service inside the Autonomous Agent service. It builds each call's session (instructions, tools, voice, speech detection, language) and holds a server-side control connection to the live call — the sideband — through which it receives transcripts and tool calls, and tells the model what to say. It owns the call's silence timers, hold clock and language. The model's tool surface is deliberately small:
  • start_conversation and continue_conversation — send the caller's request to the master orchestrator
  • cancel_current_task — stop the request in progress when the caller asks
  • decide_pending_request — when a new request arrives while another is running, act on the caller's choice to switch now or after
  • set_call_language — move the call to the language the caller is speaking
  • end_call — only on an explicit goodbye
  • transfer_to_human — phone calls only, and only when a transfer number is configured
• Master orchestrator (Leena Autonomous Agent): unchanged from chat — planning, tool selection, execution, knowledge grounding and long-running job coordination. Every substantive request is dispatched to it as a normal run, and the full answer also lands in the employee's chat thread.
• Telephony: phone numbers sit on a Leena AI Twilio Elastic SIP trunk whose origination points at OpenAI's SIP endpoint. Leena AI places the number on the trunk when phone calls are enabled; no SIP details are entered by the admin.
• State layer: live call state is kept in Redis so any pod can read it. Each call's sideband runs on exactly one pod, its owner; commands that land on another pod (hold, hang up, keypad input) are routed to the owner. A pod shutting down ends its live calls cleanly first. The final transcript is archived to MongoDB.

Call flow

How a web call and a phone call are set up, what happens on each turn, and how the call closes.

sequenceDiagram
    autonumber
    participant C as Caller<br/>(browser or phone)
    participant O as OpenAI<br/>GPT Realtime
    participant G as Leena AI Core<br/>(Gateway)
    participant R as Voice runtime
    participant M as Master<br/>orchestrator

    Note over C,M: Web call setup
    C->>G: Start call
    Note right of G: Verify caller,<br/>one-call guard
    G->>R: Build session (instructions, tools, voice, language)
    C->>G: WebRTC offer
    G->>R: Relay offer
    R->>O: Create call
    O-->>C: Answer (relayed back via Core)
    C->>O: Audio flows directly, both ways
    R->>O: Attach sideband (control channel)

    Note over C,M: Phone call setup
    C->>O: Dial number (Twilio SIP trunk to OpenAI SIP)
    O->>G: Signed incoming-call webhook
    Note right of G: Verify signature, resolve assistant,<br/>gates, OTP or KBA
    G->>R: Accept call
    R->>O: Accept and attach sideband

    Note over C,M: Each turn
    C->>O: Speaks
    O->>C: Short acknowledgement
    O->>R: start or continue conversation
    R->>M: Dispatch request
    M-->>R: Progress steps
    R->>O: Speak progress update
    M-->>R: Answer
    R->>O: Speak answer
    O->>C: Spoken answer
    R->>G: Transcript turn and cards
    G->>C: Cards in chat (SMS on phone)

    Note over C,M: Call end
    R->>G: Call ended (transcript, usage, end reason)

Request lifecycle

• Web call start: The client requests a call from Core with the employee's session token. Core verifies the caller before minting any credentials, rejects a second concurrent call for the same user, creates a call ID and short-lived call-scoped credentials, and asks the voice runtime to build the session. The client must acknowledge the call within a few seconds or it is cancelled cleanly.
• Media setup (web): The browser's WebRTC offer is relayed through Core and the voice runtime to OpenAI, and the answer comes back the same way. From then on, audio flows directly between the browser and OpenAI. OpenAI credentials never reach the browser. Once the connection is up, the voice runtime attaches its sideband.
• Phone call start: The caller dials the assistant's Twilio number, which reaches OpenAI over the SIP trunk. OpenAI sends a signed incoming-call webhook to Core, which:
  • Verifies the signature (HMAC-SHA256, with a five-minute timestamp tolerance) and ignores duplicate deliveries
  • Resolves the assistant from the dialled number, and the caller from their number
  • Checks that phone calls are enabled, the assistant is on voice 2.0, the kill switch is off, and the caller is within the spam limit (by default, 5 calls per 15 minutes per caller)
  • Refuses at the SIP layer when a check fails, or accepts the call and asks the voice runtime to answer it
• Phone-call authentication: Callers are identified by OTP or knowledge-based authentication (KBA) on the keypad, with the same rules and attempt limits as before. OpenAI reports each keypress, and Core collects the digits until the code is complete or the caller presses #. When identification fails, the assistant answers and explains why rather than dropping the call silently.
• Greeting: On web calls, the assistant greets the employee in the language they use the web app in, or the assistant's default language. On phone calls, it speaks the assistant's default language until the caller is identified — OTP and KBA prompts included — then the employee's language.
• Turns: The speech-to-speech model hears the caller, gives a short acknowledgement, and sends the request — in the caller's own words — to the master orchestrator. Simple conversational turns (greetings, confirmations, clarifications) are answered by the model directly.
• Progress updates: The orchestrator streams its real steps ("Fetching leave balance"), and the model speaks them when the caller would otherwise be waiting in silence. Once the answer has been spoken, no more progress updates follow for that turn.
• Response delivery: The orchestrator's own answer is spoken once, as written — not paraphrased by the voice model.
• Rich content: Anything that can't be spoken cleanly — links, tables, citations — lands in the chat window as cards alongside the transcript. On phone calls, cards are also sent by SMS.
• New requests mid-task: If the caller asks for something different while a request is still running, the new request is held and the assistant asks whether to stop the current one or take the new one after it. "Cancel that" stops the running request.
• Barge-in: The caller can interrupt at any time; the model stops speaking and listens. Turn ends are detected from the audio using the assistant's speech detection threshold.
• Language switching: The speech-to-speech model decides the language from what it hears; there is no separate language detector and no per-assistant language list. When the caller speaks a full sentence in another language, the voice runtime switches the call to it — speech, caller transcription, progress updates and the goodbye all follow, and the request is re-tagged with the new language so the orchestrator answers in it. Single words, names, numbers, greetings and thanks don't switch the call, and a sentence that mixes languages counts as the one it is mainly in. Any language Leena AI supports is accepted, and a caller can correct a wrong pick by saying which language they're speaking. Nothing is remembered per employee between calls.
• Mid-call sign-in: When a workflow needs the employee to sign in (an OAuth prompt, for example), the call waits for it with periodic spoken updates, then resumes the request.
• Hold and resume: Client-initiated and broadcast to every open tab. A hold takes effect only once the voice runtime confirms it. Time on hold is excluded from the chat transcript.
• Human transfer (phone only): When the caller asks for a person, the assistant says one hand-off line and the call is transferred by SIP to the configured transfer number. Web calls can't be transferred.
• Auto termination: A call ends automatically in three cases, and the agent always says why before the line drops:
  • About 12 seconds of silence prompts a check-in; the call ends if there's still no reply 20 seconds later
  • The call reaches the maximum duration (15 minutes)
  • A hold exceeds 5 minutes — with a spoken warning 30 seconds before disconnect
• Resilience: Every event from the voice runtime to Core carries a sequence number and is acknowledged; unacknowledged events are retried a bounded number of times, and duplicates are skipped. Because call state lives in Redis, any pod can serve a request for a live call.
• Observability: Per call, Leena stores the transcript, end reason, greeting and spoken languages, token usage and cost, and the request IDs the call dispatched, in the customer's hosting region. Calls on voice 2.0 appear under Call history in AI Colleagues Studio, with status, disconnection reason and the runs each call started. Call recording playback there is off unless Leena AI turns it on for the assistant.

Latency design (why it feels instant)

• One model, not three: speech-to-speech removes the hand-offs between speech recognition, a language model and speech synthesis.
• Direct media path: audio flows directly between the caller and OpenAI; Leena never relays it.
• Instant acknowledgement: the voice model acknowledges each request immediately, while the orchestrator works.
• Progress paced by silence: progress updates are spoken when the caller would otherwise hear nothing, so long tasks don't feel dead.
• Orchestrator unchanged: the master orchestrator keeps its context limits, selective retrieval and caching, so voice inherits chat's execution speed.

Dialogue quality

• Native voices: the model's own voices, with tone and speaking speed shaped by the admin's voice model instructions.
• Natural turn-taking: interruptions and end-of-turn detection are handled in the model, from the audio itself.
• Noise reduction: tuned for handsets on phone calls and for laptop microphones on web calls.
• Transcription in the call's language: caller transcripts follow the current call language, so speech in one language isn't written out in another script. The one sentence that triggers a switch is transcribed under the previous language, so it can appear transliterated in the transcript.
• One voice, every language: each voice speaks whichever language the call is in, so switching language doesn't change how the assistant sounds.
• Spoken goodbye: the call ends only on an explicit goodbye, and the closing line is spoken in the call's language.

Security, privacy, and data residency

• Transport security: Web call audio uses WebRTC, which is always encrypted. Control traffic between Core, the voice runtime and OpenAI uses HTTPS and WSS (TLS).
• Credentials: OpenAI keys stay on the server. The browser receives only a call-scoped token, signed with a key separate from login tokens; the runtime's credentials for each call expire after one hour.
• Signed phone admission: incoming phone calls are admitted only after Core verifies OpenAI's webhook signature.
• Availability control: An admin-controlled kill switch disables an assistant instantly; incoming phone calls are refused rather than left to ring.
• Logs: service logs record the length of what was said, not the words. Transcripts reach Core as chat history, under Core's retention.
• Hosting region: Leena persists transcripts and recordings in the same region as the customer's Leena tenant. Live call state lives in Redis and expires after the call.
• Model region: voice 2.0 uses a dedicated OpenAI project for each Leena hosting cluster. EU clusters use an EU-resident OpenAI project. For other regions, confirm with your Leena AI contact.

Voice 1.0 (deprecated)

Voice 1.0 remains in service for assistants that haven't switched, until it is retired by January 2027.

Architecture overview

Call audio travels between the employee and Retell.ai, which turns speech into text for Leena AI and turns Leena AI's replies back into speech.

flowchart TB
    subgraph CALLERS["Employees"]
        direction LR
        WEB["Web app<br/>(browser)"]
        PH["Phone caller"]
    end

    subgraph MEDIA["Call audio"]
        direction LR
        RTL["Retell.ai (US hosted)<br/>speech recognition, speech synthesis,<br/>session, telephony"]
        TW["Your Twilio number<br/>+ SIP trunk"]
    end

    subgraph LEENA["Leena AI platform (your hosting region)"]
        direction LR
        CORE["Leena AI Core<br/>(Gateway)"]
        VO["Voice Orchestrator<br/>(fast path)"]
        MO["Master orchestrator"]
        TOOLS["Tools, AOPs, knowledge,<br/>connected systems"]
        STORE[("Redis + MongoDB<br/>call state, transcripts")]
    end

    WEB == "audio (WSS)" ==> RTL
    PH == "audio" ==> TW
    TW == "SIP" ==> RTL

    WEB -- "start call, cards in chat" --- CORE
    RTL -- "transcripts in, text to speak out (WSS)" --- CORE
    CORE -- "turns, spoken replies" --- VO
    VO -- "complex turns" --> MO
    MO --> TOOLS
    CORE --- STORE

Key components

• Retell.ai: session management, real-time ASR, and TTS. Provides interruption/barge-in signals, streams transcripts to Leena AI, and carries the telephony leg (including SIP transfer) for phone-based deployments.
• Leena AI Core (Gateway): owns the Retell session and everything around the call — call initiation and tokens, the bot kill switch, the one-active-call-per-user guard, transcript assembly into chat history, hold state, rich-content delivery (chat or SMS), phone-call authentication, and call analytics.
• Voice Orchestrator (perimeter fast path): a low-latency mini-agent running inside the Autonomous Agent service. It is deliberately small — temperature 0, a short output budget, and a four-tool surface. It:
  • Instantly acknowledges or clarifies
  • Provides micro-responses to small talk and confirmations
  • Gates and forwards complex turns to the core agent (converse_with_main_agent)
  • Controls the session: cancel a running task, end the call, or request a human transfer
  • Rewrites core-agent output into speech-friendly language (strips markdown, turns lists into spoken transitions)
• Core Orchestrator (Leena Autonomous Agent): planning, tool selection, execution, knowledge grounding, and long-running job coordination.
• Model routing: both orchestrators resolve their model through Leena's provider-agnostic LLM layer rather than a pinned model name — Azure OpenAI primary with OpenAI failover on the voice fast path — so models can be upgraded without a code change.
• State layer: live call state is held in memory on the pod owning the WebSocket, snapshotted to Redis (2-hour TTL) and broadcast over Redis Pub/Sub so any pod can recover a call; the final transcript is archived to MongoDB.
• Text-to-Speech (TTS): Retell.ai primary; voice persona per customer.

Call flow

Request lifecycle

• Call start: The client requests a call from Core, which checks the bot kill switch and rejects a second concurrent call for the same user. Core returns a call token; the client opens a secure WebSocket (WSS) to Retell.ai and audio streams bi-directionally.
• Greeting: The first turn (response_id 0) returns the configured greeting without a full model pass. The client keeps the microphone muted through the greeting to avoid speaker-to-mic echo, then unmutes automatically.
• Real-time ASR: Retell.ai emits partial/final transcripts to Leena over WSS (encrypted in transit).
• Language: voice 1.0 is English only. Speech recognition is set to US English (en-US) for web and phone calls alike, and the web app labels the call button Start voice chat (English only).
• Perimeter fast-ack: The Voice Orchestrator consumes streaming text and issues an instant, sentiment-matched acknowledgement (e.g. "Got it — checking your PTO") without blocking on the core agent.
• Routing:
  • Simple/safe turns (greetings, confirmations, clarifications) are answered by the Voice Orchestrator directly
  • Turns requiring Knowledge/AOP/Tools go to the Core Orchestrator
• Execution (Core): intent grounding → plan → tool/API calls (e.g. HRIS). Context is kept minimal (see latency section) and cached when safe.
• Progress updates: The core agent streams machine-readable status ("checking KB", "calling Time-Off API"), which the Voice Orchestrator rephrases as natural speech. Filler updates escalate over the wait — short holds first, then reassurance, then an acknowledgement of complexity. If the core agent is blocked on the user (an OAuth sign-in, for example), the fillers switch to nudges instead of reassurance.
• Response delivery: Text leaves the orchestrator in chunks marked incomplete, and Core forwards each chunk to Retell.ai for synthesis as it arrives — so audio begins playing while the rest of the answer is still being written. Acknowledgements, progress updates, and final answers all travel this same streaming path.
• Rich content: Anything that cannot be spoken cleanly — links, tables, citations — is emitted as a separate chat message. On web calls it lands in the chat window alongside its sources; on phone calls it is delivered as an SMS. The chat copy is treated as canonical for that turn, so the spoken transcript is not persisted twice.
• Barge-in/interrupts:
  • Retell.ai signals user interrupt events mid-utterance and issues a new response_id
  • The orchestrator applies "latest turn wins": the in-flight stream is aborted and any running core-agent task is gracefully stopped before the new turn proceeds
  • Core reports back which portion of the previous answer was actually played, and the orchestrator tags the transcript accordingly — so it neither repeats what the caller already heard nor assumes they heard something that was cut off
• Mid-call actions: Hold and resume are client-initiated and broadcast to every open tab; hold state is persisted to Redis so it survives a pod restart, and time spent on hold is excluded from the chat transcript. Files uploaded from the chat composer are registered against the live call, acknowledged out loud on the open turn, and forwarded to the core agent with the current request.
• Human transfer (phone only): When the orchestrator requests a handover, Core validates the configured E.164 transfer number at dispatch time and Retell.ai performs the SIP transfer after the hand-off line is spoken. Web calls cannot be SIP-transferred.
• Phone-call authentication: Inbound callers are identified by phone number, then verified with either an OTP entered over DTMF or knowledge-based authentication (two mandatory identifiers plus a tiebreaker, matched against a hashed lookup), with per-step and per-flow retry limits.
• Observability: Per call, Leena stores metadata, audio, transcripts (chronological), and summaries for analytics and debugging, in the customer's hosting region. Call lifecycle events — initiated, connected (with failure reason), disconnected (with end type) — are emitted to the analytics pipeline.
• Resilience: Model and thread-dispatch failures retry with exponential backoff; if a WebSocket reconnects to a different pod, call state is rebuilt from memory → Redis → MongoDB.
• Auto Termination: A call ends automatically in three cases, and the agent always says why before the line drops:
  • No activity for ~20 seconds after the agent finishes speaking
  • The call exceeds the maximum duration (15 minutes)
  • A hold exceeds 5 minutes — with a spoken warning 30 seconds before disconnect

Latency design (why it feels instant)

We achieve near-instant acks (less than a human pause), smooth turn-taking, and fast task completion via:

• Streaming out: spoken text is emitted and synthesised chunk by chunk, so time-to-first-audio is set by the first few tokens rather than the full answer.
• Perimeter fast path: a small, fast model with a short output budget produces immediate, context-aware acks and simple replies.
• Streaming ASR: text tokens arrive as the user speaks; no end-of-speech blocking.
• Shallow turn path: the route from transcript to synthesised audio carries no redundant model passes between the orchestrator's decision and speech.
• Progressive disclosure: the user hears meaningful updates while the core agent executes.
• Context diet: the core agent strictly limits prompt/context size per turn; it uses selective retrieval and memoised tool results (caching) where permissible.
• Parallelism: plan and tool-prep run in parallel with TTS buffering of non-critical preambles; heavy events are dispatched to a task pool so the WebSocket never blocks on model latency, while heartbeats and transcript updates are handled inline.

Dialogue quality: prosody, sentiment, and control

• Naturalness default: modern TTS voices are humanlike even without SSML.
• Production control: for long or sensitive reads (spelling, reset steps), we apply SSML (pace, pauses, emphasis, repeat) on top of vendor defaults.
• Sentiment alignment: the Voice Orchestrator infers user sentiment and mirrors tone by selecting from acknowledgement families — neutral, upbeat, empathetic, supportive, appreciative — and by choosing phrasing and SSML cues to match.
• Voice-first rewriting: core-agent output is written for screens, so it is passed through a voice-specific rewrite before synthesis: markdown removed, lists converted into spoken transitions, length trimmed, and anything unspeakable diverted to chat or SMS.

Security, privacy, and data residency

• Transport security: All hops use WSS (TLS), bi-directional.
  • Client to Retell.ai (audio up)
  • Retell.ai to Leena (transcripts)
  • Leena to TTS to Retell.ai (audio down)
• Availability control: An admin-controlled kill switch disables an assistant instantly; incoming calls are rejected with a spoken message rather than failing silently.
• Retention controls at the subprocessor: Retell.ai is configured not to persist call data beyond a minimal operational window; Leena stores authoritative logs.
• Hosting region: Leena persists audio/transcripts in the same region as the customer's Leena tenant. Live call state lives in Redis with a 2-hour TTL and is evicted when the call ends.
• Constraint: Retell.ai currently hosts in US regions only; this can block strict-localisation customers (e.g. some ME countries, EU-only mandates). Voice 2.0 doesn't use Retell.ai — see its security, privacy, and data residency section.