Skip to content
NJNakshatra Joshi
← All work
Product/2025

Voice AI Agent

Sub-second mic → STT → LLM → TTS loop, provider-swappable by env var

mic→STT→LLM→TTS→playback
4
stages in the loop
1
env var to swap providers
3
typed upstream failure modes

The problem

A voice loop is four network calls deep, and each one can rate-limit, stall or fail. Latency is the product: anything the user can perceive as waiting breaks the illusion of conversation.

Approach

01

One interface, two providers

Groq and OpenAI sit behind a single provider abstraction selected by one env var, so the whole stack can be swapped when one is rate-limited or a model is deprecated — no code change, no redeploy of a different build.

02

Degrade, don't die

Browser speech synthesis is the fallback when TTS is unavailable, and rate-limit-aware sentence trimming keeps the agent inside quota rather than failing a long response outright.

03

Type the failures

429, 502 and 503 are distinct, typed error paths reaching the client, because 'try again in a moment' and 'the model is gone' need different UI.

Outcome

  • A conversational loop that survives provider rate limits instead of dropping the session.
  • Provider migration is a config change, which made comparing Groq and OpenAI latency a five-minute experiment.

What I'd do differently

Streaming the TTS rather than waiting for the full clip is the single biggest remaining latency win, and the architecture is already shaped for it.