Voice AI Agent
Sub-second mic → STT → LLM → TTS loop, provider-swappable by env var
- 4
- stages in the loop
- 1
- env var to swap providers
- 3
- typed upstream failure modes
The problem
A voice loop is four network calls deep, and each one can rate-limit, stall or fail. Latency is the product: anything the user can perceive as waiting breaks the illusion of conversation.
Approach
One interface, two providers
Groq and OpenAI sit behind a single provider abstraction selected by one env var, so the whole stack can be swapped when one is rate-limited or a model is deprecated — no code change, no redeploy of a different build.
Degrade, don't die
Browser speech synthesis is the fallback when TTS is unavailable, and rate-limit-aware sentence trimming keeps the agent inside quota rather than failing a long response outright.
Type the failures
429, 502 and 503 are distinct, typed error paths reaching the client, because 'try again in a moment' and 'the model is gone' need different UI.
Outcome
- A conversational loop that survives provider rate limits instead of dropping the session.
- Provider migration is a config change, which made comparing Groq and OpenAI latency a five-minute experiment.
What I'd do differently
Streaming the TTS rather than waiting for the full clip is the single biggest remaining latency win, and the architecture is already shaped for it.