How to Build a Voice Agent with LangChain: Architecture, Streaming, Tools, and Production Patterns
Building a voice agent is not simply a matter of connecting speech-to-text to an LLM and adding text-to-speech.
A production voice agent has to solve a harder problem:
How do you make an AI system listen, reason, use tools, remember context, and respond quickly enough that the conversation still feels natural?
LangChain can handle the agent and tool-orchestration layer, but the realtime experience depends heavily on what happens around it.
A practical architecture looks like this:
User microphone
↓
Audio streaming
↓
Speech-to-Text (STT)
↓
Transcript / turn detection
↓
LangChain Agent
↓
Tools / APIs / Business Logic
↓
Streaming response
↓
Text-to-Speech (TTS)
↓
User hears response
LangChain's current voice-agent documentation describes this as the "sandwich" architecture: STT → agent → TTS. The advantage is that each layer can be replaced independently, while the agent can continue using the broader LangChain ecosystem.
What You Actually Need to Build
Before writing code, separate the voice agent into five responsibilities:
- Audio transport — moves microphone audio to the backend and audio responses back to the client.
- Speech recognition — converts audio into text.
- Agent reasoning — decides what the user wants and what action to take.
- Tool execution — interacts with databases, CRMs, calendars, APIs, or internal systems.
- Speech synthesis — converts the response back into audio.
This separation matters because these components have different performance characteristics.
For example, changing your TTS provider should not require rewriting your business logic. Similarly, changing the LLM should not require rebuilding your audio transport.
That modularity is one of the strongest reasons to use a cascaded architecture instead of putting everything into one model.
1. Choose the Voice Architecture First
There are two major ways to build a voice agent.
Architecture A: STT → Agent → TTS
Audio
↓
STT
↓
Text
↓
LangChain Agent
↓
Text
↓
TTS
↓
Audio
This gives you control over every component.
You can choose one STT provider, another LLM, and a completely different TTS provider.
It also makes debugging easier because you can inspect the transcript, agent decision, tool call, and final response independently.
The trade-off is additional infrastructure and potential latency.
Architecture B: Speech-to-Speech
Audio
↓
Multimodal Voice Model
↓
Audio
This can reduce the number of moving pieces and can preserve more information about how something was spoken, such as tone.
However, it can reduce your control over individual components and introduce provider-specific constraints.
For business applications where tool execution, observability, provider flexibility, and deterministic workflows matter, the cascaded architecture remains highly practical.
2. Use Streaming Instead of Waiting for Complete Responses
This is where many voice-agent implementations go wrong.
A naive implementation waits for the entire chain:
Record entire sentence
↓
Transcribe
↓
Wait for complete LLM response
↓
Generate complete audio
↓
Play response
The user experiences one long delay.
A streaming architecture instead looks like:
Audio chunk
↓
STT starts immediately
↓
Transcript arrives
↓
Agent starts generating
↓
First response tokens arrive
↓
TTS starts
↓
Audio starts playing
The system does not wait for every stage to finish before the next stage begins.
LangChain's official voice-agent example uses asynchronous streaming and RunnableGenerator to connect STT, the agent, and TTS. The documentation notes that this pipeline can achieve sub-700 ms latency with suitable STT and TTS providers.
The important lesson is:
Realtime voice is primarily a pipeline-design problem, not just a model-selection problem.
Research on realtime voice agents similarly identifies streaming and pipelining across STT, LLM, and TTS as a central mechanism for reducing perceived latency.
3. Create the LangChain Agent
Once speech has been converted into text, the voice layer can hand the request to a normal LangChain agent.
Current LangChain applications use create_agent as the primary entry point.
A minimal agent can look like this:
from langchain.agents import create_agent
def check_order_status(order_id: str) -> str:
"""Return the current status of an order."""
return f"Order {order_id} is currently being processed."
agent = create_agent(
model="openai:gpt-5.4",
tools=[check_order_status],
system_prompt="""
You are a customer support voice agent.
Keep spoken responses short.
Ask for missing information instead of guessing.
Use tools whenever the user asks for account-specific information.
"""
)
The important part is not the five lines of code.
It is the tool boundary.
A voice agent should not directly manipulate your database or business systems through arbitrary model-generated text.
Instead:
User:
"Where is order 4821?"
↓
Agent
↓
check_order_status("4821")
↓
Business system
↓
Structured result
↓
Agent
↓
"Your order is currently being processed."
LangChain agents can reason over available tools and execute them as part of the agent loop. The current agent implementation is built on LangGraph's runtime.
4. Design Tools for Voice, Not Just for Chat
This is an overlooked part of voice-agent engineering.
A tool that works well for a text chatbot may be poorly designed for a voice agent.
For example, avoid giving the agent a tool that returns:
{"customer_id": 1827,
"subscription_status": "active",
"plan": "enterprise",
"billing_cycle": "annual",
"last_payment": "...",
"payment_method": "..."}
if the only thing the user asked was:
"Is my subscription active?"
Instead, make the tool return information that the agent can quickly reason over.
def get_subscription_status(customer_id: str) -> str:
"""Check whether a customer's subscription is active."""
...
The voice agent can then respond:
"Yes, your subscription is active."
The rule is simple:
Design tools around decisions, not database tables.
This reduces unnecessary reasoning and makes spoken responses easier to control.
5. Keep Spoken Responses Short
A language model optimized for written chat can produce paragraphs.
A voice agent should not.
Compare:
Chatbot response:
"Certainly. I can help you with that. According to the information available in your account, your order has been processed successfully and is currently in transit. You can expect delivery within the next two to three business days..."
Voice response:
"Your order is in transit. It should arrive within two to three business days."
Voice requires a different response policy.
A useful system instruction is:
You are a voice assistant.
Speak naturally and concisely.
Prefer one or two sentences per response.
Do not read JSON, URLs, IDs, tables, or long lists aloud.
Ask one question at a time.
If a tool fails, explain the problem briefly and offer the next action.
Never invent information that is unavailable from a tool.
This is not merely prompt optimization.
It is interface design.
6. Add Conversation Memory Carefully
Voice conversations become awkward if the agent forgets what was said five seconds earlier.
Consider:
User: "I want to book an appointment tomorrow."
Agent: "What time?"
User: "Around 4."
Please ensure the agent understands that "4" refers to the appointment.
LangChain's voice-agent example uses conversation state with a checkpointer and a unique thread ID so the agent can retain context across turns.
Conceptually:
User
↓
Voice session ID
↓
Conversation state
↓
LangChain agent
↓
Response
For a production system, distinguish between:
Short-term conversation state
Things said during the current call.
Examples:
- user's name
- requested appointment time
- current order number
- selected product
Long-term business memory
Information that should survive the call.
Examples:
- customer preferences
- previous interactions
- account information
Do not put every piece of customer data into the LLM's conversation history.
Retrieve what is needed for the current decision.
7. Handle Interruptions
This is one of the biggest differences between a chatbot and a voice agent.
Imagine the agent is saying:
"Your appointment is scheduled for Thursday at—"
The user interrupts:
"Actually, make that Friday."
A real voice interface should stop speaking.
That means your system needs to support barge-in.
A simplified flow is:
Agent speaking
↓
User starts talking
↓
Detect interruption
↓
Stop TTS playback
↓
Cancel/ignore remaining audio
↓
Process new user input
Without interruption handling, the system feels less like a conversation and more like an IVR reading a script.
This is why audio transport, turn detection, and cancellation logic are just as important as the LLM.
8. Use WebSockets for Browser-Based Streaming
For a browser-based implementation, WebSockets are a practical transport layer.
The client captures microphone audio:
Browser microphone
↓
PCM audio chunks
↓
WebSocket
↓
Backend
The backend sends synthesized audio back through the same connection:
Backend
↓
TTS audio chunks
↓
WebSocket
↓
Browser
↓
Speaker
LangChain's reference voice application uses WebSockets for bidirectional audio streaming and notes that the same general architecture can be adapted to telephony or WebRTC.
The important design decision is to keep the transport layer independent from the agent.
Your agent should not care whether the request came from:
- a browser
- a mobile application
- a phone call
- a WebRTC client
It should receive an input event and return agent events.
9. Connect the Pieces with an Async Pipeline
A simplified LangChain pipeline can conceptually look like:
from langchain_core.runnables import RunnableGenerator
pipeline = (
RunnableGenerator(stt_stream)
| RunnableGenerator(agent_stream)
| RunnableGenerator(tts_stream)
)
Each stage consumes and produces a stream.
STT events
↓
Agent events
↓
TTS events
This is more useful than treating the voice agent as one giant function.
Each stage can be measured independently.
For example:
Audio received
↓
STT first transcript 180 ms
↓
Agent first token 220 ms
↓
TTS first audio 160 ms
↓
User hears response ~560 ms
These measurements tell you where the actual bottleneck is.
10. Measure the Right Latency
Do not measure only total API response time.
For voice systems, track at least:
Time to first transcript
How quickly does the system understand the user's speech?
Time to first token
How quickly does the agent begin responding?
Time to first audio
How quickly does the user hear the response?
Total response duration
How long does the agent take to finish speaking?
Tool latency
How long do external API calls take?
For example:
User finishes speaking
│
├── STT: 210 ms
│
├── Agent starts: 35 ms
│
├── CRM API: 420 ms
│
├── LLM first token: 180 ms
│
└── TTS first audio: 140 ms
If your first-audio latency is 1.2 seconds, changing the LLM may not solve the problem if the real bottleneck is a 700 ms CRM API.
11. Make External Tools Fast
Voice agents expose slow backend systems immediately.
Imagine:
Voice input
↓
Agent
↓
CRM
↓
Database
↓
Payment API
↓
Agent
↓
TTS
Even if your LLM is extremely fast, the conversation can feel slow because of downstream services.
Use:
- timeouts
- retries where safe
- caching
- parallel API requests where possible
- lightweight tool responses
- asynchronous execution
For example, if the agent needs customer information and appointment availability, those lookups may not always need to happen sequentially.
But be careful with parallel execution when tools have side effects.
Reading two systems in parallel is very different from creating two appointments simultaneously.
12. Add Failure Handling Before Production
Voice agents fail differently from chatbots.
Potential failures include:
- STT misses a word
- user speaks over the agent
- network drops
- TTS fails
- tool times out
- LLM generates an invalid tool argument
- user changes their request halfway through
- external API returns incomplete data
The agent should have explicit fallback behavior.
For example:
Tool timeout
↓
Retry if operation is safe
↓
Still failing?
↓
Tell the user
↓
Offer alternative action
Never let the model hide a failed transaction by pretending it succeeded.
For actions such as payments, bookings, cancellations, or account changes, the system should verify the actual backend result before confirming completion.
13. Where LangChain Helps — and Where It Doesn't
LangChain is useful for:
- agent orchestration
- tool calling
- model abstraction
- conversation state
- streaming agent output
- integrating business tools
- connecting the agent to LangGraph-based workflows
But LangChain is not your complete voice infrastructure.
You still need to solve:
- microphone capture
- audio encoding
- WebSockets/WebRTC
- speech recognition
- voice synthesis
- interruption handling
- latency management
- telephony integration, if applicable
- production monitoring
Think of LangChain as the reasoning and orchestration layer, not the entire voice stack.
14. When LangGraph Becomes Important
A simple voice assistant might only need:
User → Agent → Tool → Response
A business workflow can become more complicated:
Incoming call
↓
Identify customer
↓
Understand intent
↓
Check account
↓
Determine eligibility
↓
Call external system
↓
Human approval?
↙ ↘
Yes No
↓ ↓
Human Complete
review
This is where graph-based orchestration becomes valuable.
LangChain's current create_agent implementation already uses LangGraph underneath, while direct LangGraph workflows are useful when you need more explicit control over state, branching, persistence, interrupts, or complex workflows.
The important point is:
Do not add LangGraph simply because you are building a voice agent. Add graph-level orchestration when the workflow actually needs it.
15. A Production Voice Agent Architecture
A practical production architecture could look like this:
┌──────────────────┐
│ Web / Mobile │
│ / Phone Client │
└────────┬─────────┘
│
Audio Stream
│
▼
┌──────────────────┐
│ Audio Gateway │
│ WebSocket/WebRTC │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ STT │
└────────┬─────────┘
│
Transcript
│
▼
┌──────────────────┐
│ LangChain Agent │
│ │
│ State + Tools │
└───────┬──────────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
CRM/API Database Calendar
│ │ │
└───────────┼───────────┘
│
▼
Agent Response
│
▼
┌──────────────────┐
│ TTS │
└────────┬─────────┘
│
▼
Audio Stream
│
▼
User
This architecture has an important property:
Every layer can evolve independently.
You can replace the STT provider without rebuilding the agent.
You can replace the LLM without rebuilding the audio gateway.
You can replace the CRM without changing the voice interface.
That is what makes the architecture suitable for production.
Conclusion
Building a voice agent with LangChain is not primarily about writing an LLM prompt.
The difficult engineering work is around the LLM:
- streaming audio
- reducing time-to-first-audio
- managing conversation state
- designing voice-specific tools
- handling interruptions
- controlling external API latency
- validating side effects
- recovering from failures
- monitoring complete conversations
LangChain gives you a strong agent and tool-orchestration layer, while the voice infrastructure handles the real-time UI interface. Its current documentation demonstrates this separation through a streaming STT → LangChain agent → TTS architecture.
If you’re planning to take this architecture beyond a prototype and build a production-ready voice system with custom workflows, backend integrations, multilingual support, monitoring, and low-latency interactions, explore Ciphernutz’s AI Voice Agent Development services.
The goal is not to make an LLM speak. The goal is to make a business workflow conversational without making it unreliable.
