Building a Voice Agent for a Local Indian Store: 10 Days of Voice Agents — VoiceForBharat Edition

Building a voice agent sounds simple at first.

Listen to the user, send the text to an LLM, generate a response, and speak it back.

But once you start adding real-world requirements — memory, safety guardrails, multilingual conversations, phone calls, human escalation, analytics, and specialist handoffs — it becomes a very different engineering problem.

Over the last 10 days, I worked on exactly that as part of 10 Days of Voice Agents — VoiceForBharat Edition. I started with a basic voice agent and gradually turned it into a voice assistant designed for a local Indian store.

Here's what I built and what I learned along the way.

The Problem I Wanted to Solve

For a local grocery or general store, customers often have simple questions:

• Is a particular product available?
• What is the price?
• What are the store timings?
• Are there any offers?
• Can I get an alternative product?
• What should I do if my product is damaged or incorrect?

These questions don't always need a person to answer them. I wanted to build a voice-first assistant that could handle these conversations naturally, while also knowing when it should stop trying to solve the problem itself.

Voice is particularly useful here because customers don't have to open an app, type their question, or navigate through menus. They can simply speak.

Meet the Voice Agent

My project is a voice assistant for a local Indian grocery/general store. It's designed to:

• Help customers with shopping-related questions
• Handle English and Hindi conversations
• Understand Hinglish-style conversations
• Remember returning customers
• Follow safety and accuracy guardrails
• Hand off return and refund requests to a specialist agent
• Make outbound phone calls
• Track call outcomes
• Provide analytics on how the agent is performing

The goal wasn't just to make an AI that could talk. It was to make the conversation behave like a useful customer-service system.

How the System Works

The basic voice pipeline looks like this:

User speaks
    ↓
Deepgram Speech-to-Text
    ↓
Google Gemini LLM
    ↓
Tools / Memory / Agent Logic
    ↓
Murf Falcon TTS
    ↓
LiveKit real-time audio
    ↓
User hears the response

• LiveKit handles the real-time communication between the user and the agent.
• Deepgram Nova-3 handles speech recognition.
• Google Gemini handles language understanding and response generation.
• Murf Falcon, using an Indian English voice, handles text-to-speech.

The agent session also uses multilingual turn detection, voice activity detection, preemptive generation, and noise cancellation to make the conversation feel more natural.

An Indian Voice With Murf Falcon

One of the most important parts of this project was the voice itself.

I used Murf Falcon with an Indian English voice and configured the agent to switch between en-IN and hi-IN depending on the conversation. The project detects Hindi/Hinglish-style input and adjusts the TTS locale accordingly, so the interaction isn't restricted to formal English — a user can speak in English, Hindi, or Hinglish, and the agent responds in the same conversational style.

For this challenge, I used Murf Falcon — the fastest TTS API — as the text-to-speech layer of the agent.

Memory for Returning Customers

A useful assistant shouldn't treat every conversation as if it's meeting the customer for the first time.

I added caller memory using SQLite. The agent has two tools:

lookup_caller()
save_caller_memory()

When a caller is recognized, the agent looks up previously saved information. If the customer shares useful preferences, the agent asks for permission before saving them.

This was an important design decision — the agent shouldn't silently collect information just because it can. The system prompt explicitly prevents the agent from inventing memories or exposing internal database information.

Guardrails and Agent Personality

Another important part of the project was defining what the agent should and shouldn't do. The assistant has clear objectives and guardrails. For example, it should never:

• Make up product prices
• Claim something is in stock without knowing
• Promise an unconfirmed discount
• Confirm an order that hasn't actually been confirmed
• Answer unrelated requests
• Provide harmful or illegal assistance

If it doesn't know something, it's instructed to say so instead of guessing. This might seem simple, but good voice agents need clear boundaries just as much as they need good prompts.

Specialist Agent Handoff

One of the features I enjoyed building was the specialist handoff.

The main store assistant handles normal shopping questions. But if the customer needs help with returns, refunds, damaged or defective products, wrong or missing items, or return eligibility, the main agent transfers the conversation to a dedicated Returns and Refunds Specialist.

The important part is that the specialist receives the existing conversation context — the customer doesn't have to explain the problem again. The flow looks like this:

Customer
   ↓
Main Store Assistant
   ↓
Return / Refund request detected
   ↓
Returns & Refunds Specialist
   ↓
Continue existing conversation

This made the project feel much closer to a real customer-support system than a single chatbot.

Outbound Phone Calls

I also added outbound calling using LiveKit's SIP capabilities. The outbound call flow creates a unique room, dispatches the agent to that room, and creates an outbound SIP participant:

Agent
  ↓
LiveKit Room
  ↓
SIP Participant
  ↓
Phone Network
  ↓
Customer

This was one of the more challenging parts, since there are more moving pieces here than in a browser-based voice conversation. The system has to coordinate the LiveKit agent, the room, the SIP configuration, and the phone connection.

Call Analytics

Once an agent starts making calls, another question appears: how do I know whether it's actually performing well?

So I added call tracking and analytics. Each call can record:

• Call ID
• Call type
• Duration
• Intent
• Outcome
• Escalation status and reason
• Tools used

The backend exposes this data through a FastAPI service, which provides metrics like total calls, successful calls, failed calls, escalated calls, success rate, and average duration. It also supports filtering and retrieving individual call details.

This changed how I looked at the project. Instead of only asking, "Does the agent talk?" I could start asking, "Did the conversation accomplish its goal?"

The Hardest Parts

The hardest part wasn't getting the first response from the agent — it was making all the pieces work together.

Frontend
   ↓
LiveKit
   ↓
Voice Agent
   ├── Speech-to-Text
   ├── LLM
   ├── Text-to-Speech
   ├── Tools
   ├── Memory
   ├── SIP
   └── Analytics

When something goes wrong, it isn't always obvious which layer is responsible. During the challenge, I dealt with issues around running multiple services, environment variables, database state, call handling, and getting components to communicate correctly.

One lesson I learned: debugging a voice agent isn't just about debugging Python code. You also have to think about audio flow, real-time connections, API credentials, room state, SIP state, database state, and agent state.

Breaking the system into smaller components and checking each layer separately made troubleshooting much easier.

How You Can Build Your Own Voice Agent

You don't need all of these features on day one. Start with the basic pipeline:

Speech-to-Text
      ↓
     LLM
      ↓
Text-to-Speech

Then add real-time transport such as LiveKit. From there, gradually layer in:

1. A system prompt
2. Guardrails
3. Tools
4. Memory
5. Multilingual support
6. Phone calling
7. Human escalation
8. Analytics

My project is available publicly on GitHub: Codehunter0009/murf-livekit-starter

You can inspect the complete implementation, including the backend, frontend, agent logic, analytics API, database, and outbound calling code.

Basic Setup

The project uses Python for the backend and Node.js for the frontend. The backend uses uv for dependency management.

After cloning the repository:

cd backend
uv sync

Then configure the required environment variables in .env.local:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY

For outbound calling, the relevant SIP configuration is also kept in environment variables.

Never commit your .env.local file or API keys to GitHub.

Running the Agent

Start the backend in development mode:

uv run python src/agent.py dev

The project also includes a frontend for interacting with the voice agent through the browser. Once the backend, LiveKit, and frontend are running, open the application, allow microphone access, and start a conversation.

What I Learned

The biggest lesson from these 10 days is that building a voice agent is much more than connecting an LLM to a microphone. A useful voice agent needs:

Conversation + Context + Tools + Safety + Observability

The LLM is only one part of the system. I also learned how important it is to design failure paths:

• What happens when the agent doesn't know the answer?
• What happens when the customer asks for something outside its responsibility?
• What happens when a customer needs a human?
• What happens when the conversation needs a specialist?

These are the questions that turn a demo into a complete system.

What I Would Improve Next

There's plenty I'd still improve: more robust multilingual support, an expanded toolset, a better analytics dashboard, additional specialist agents, and a more reliable phone experience. I'd also like to spend more time measuring latency and real-world conversation quality, rather than evaluating the system only through individual test calls.

Project

GitHub: Codehunter0009/murf-livekit-starter

The complete source code and setup instructions are available in the repository.

Final Thoughts

Ten days ago, I was mainly thinking about how to make a voice agent talk.

By the end of the challenge, I was thinking about something much bigger: How should a voice agent behave when it doesn't know something? How should it remember users? When should it ask for permission? When should it escalate? How do we measure whether a call was successful? And how do we make the whole system reliable enough for someone to actually use?

That's what made this challenge valuable. I didn't just learn how to build a voice interface — I learned how to think about a voice agent as a complete system.

10 Days of Voice Agents — VoiceForBharat Edition completed. 🚀