Introduction
For the past 10 days, I took part in the 10 Days of AI Voice Agents — #VoiceForBharat Edition challenge.
During this challenge, I built an AI voice agent named Sadie.
My goal was not just to make an AI that could talk. I wanted to build a voice agent that could understand users, remember conversations, use tools, make phone calls, connect users to humans, and hand conversations to specialist agents.
This journey helped me understand that building a voice agent is much more than connecting an LLM with a text-to-speech API.
The Problem
Many people find it easier to speak than type.
This can be especially useful for people who want to:
Ask questions using their voice
Learn through conversation
Get quick information
Speak in Hindi or English
Use Hindi and English together
Get help without using complicated interfaces
I wanted to build a voice assistant that could make learning and getting information feel more natural.
Instead of typing a question, users can simply speak to Sadie.
What I Built
Sadie is an AI voice agent that can:
Have real-time voice conversations
Understand Hindi-English code-mixed conversations
Follow personality and safety rules
Remember information with user permission
Use external tools
Make outbound phone calls
Escalate conversations to humans
Track call information
Hand conversations to specialist agents
The specialist agents I built are:
Grammar Specialist
Maths Specialist
Full Stack Development Specialist
How the System Works
The basic architecture of my project is:
User
|
| Voice
↓
LiveKit
|
↓
Deepgram STT
|
↓
Google Gemini
/ | \
/ | \
Memory Tools Specialists
\ | /
\ | /
↓
Murf Falcon
|
↓
User Voice
Main Components
Deepgram handles Speech-to-Text.
Google Gemini acts as the brain of the agent and understands the user's request.
Murf Falcon converts the AI's response into natural speech.
LiveKit handles real-time audio communication.
I also added memory, external tools, telephony, human escalation, analytics, and specialist agents around these core components.
Day 1–3: Voice, Personality and Guardrails
The first step was making the agent speak naturally.
I configured Murf Falcon TTS with the Anisha voice.
I also created a personality and clear instructions for the agent.
One important requirement was handling Indian-style conversations and Hindi-English code mixing.
I added script rules so that:
Hindi uses Devanagari
English uses Latin characters
The agent follows its defined personality
The agent stays within its allowed tasks
This helped make the agent more consistent and reliable.
Day 4–5: Memory and Tools
Next, I added persistent memory to Sadie.
I created a SQLite database:
sadie_memory.db
The agent can store useful information about returning users, but memory storage is permission-based.
I also integrated the Free Dictionary API as an external tool.
This allowed Sadie to get useful dictionary information when required.
This taught me an important lesson:
A good AI agent should not depend only on its language model. Tools can give an agent access to useful external information.
Day 6: Outbound Phone Calls
One of the most interesting parts of the challenge was connecting Sadie to phone calls.
I configured:
LiveKit SIP
SIP trunk
Linphone
Outbound calling
The goal was to allow the voice agent to make automated tutoring calls.
This was also one of the most challenging parts of the project because telephony involves many configuration details.
Day 7: Human Escalation
An AI agent should know when it needs human help.
So I created a human escalation system.
Before escalating, the agent asks the user for permission.
After approval, the system generates a reference ID and sends the escalation information through a Discord webhook.
This creates a simple bridge between the AI and a human.
The lesson was simple:
AI should not always try to solve everything by itself. Sometimes the best action is to ask a human for help.
Day 8: Call Analytics Dashboard
After building the voice agent, I wanted to understand what was happening during calls.
So I created a real-time analytics dashboard using:
Next.js
Chart.js
SQLite
The dashboard has a floating glassmorphism design and displays call information.
This helped me move from simply building an agent to also monitoring its behavior and call outcomes.
Day 9: Specialist Agents
On Day 9, I added multi-agent functionality.
The main agent can hand a conversation to a specialist when the user's question requires a specific skill.
I created three specialists.
Grammar Specialist
Handles English grammar questions.
Maths Specialist
Handles mathematical questions and problems.
Full Stack Development Specialist
Helps with web development and full-stack development questions.
One important part was preserving the conversation context during the handoff.
The specialist should understand what the user has already discussed instead of asking the user to explain everything again.
Challenges I Faced
Building this project was not always easy.
Here are some of the biggest challenges I faced.
- Outbound SIP Calls Were Failing
When I first tested outbound calls using Linphone and LiveKit SIP, calls were dropping or timing out.
After debugging the setup, I found that the Media Encryption Mandatory setting in Linphone was causing compatibility problems with my SIP setup.
I changed the media encryption setting to Disabled/Optional, and the calls started connecting correctly.
What I learned
Telephony systems have many configuration details. Sometimes the problem is not in your code but in the configuration of the tools you are connecting.
- Incorrect Failed Calls in the Dashboard
Another problem appeared in the analytics dashboard.
When I ended a normal browser conversation, the dashboard was increasing the failed-call count.
The problem was in the logging logic. A normal browser disconnect was being treated as a failed call.
I separated browser conversations from actual phone-call outcomes and changed the dashboard to track the real phone-call results.
What I learned
Analytics are only useful when the data being recorded represents the real event.
- Specialist Handoff and Conversation Context
When I started working on specialist agents, I had another challenge.
If Sadie transferred the conversation to the Grammar, Maths, or Full Stack specialist, I needed to make sure the specialist knew what the user had already said.
I solved this by passing a copy of the existing conversation context to the specialist.
Now the specialist can continue the conversation instead of starting from zero.
What I learned
In a multi-agent system, a handoff is not only about changing the agent. Conversation context is equally important.
- Python Import Problems
While working with LiveKit background workers, I faced:
ModuleNotFoundError: No module named 'src'
The problem happened because the background process was starting from a different path.
I fixed the import path so that the worker could correctly find the project modules.
This reminded me that applications using background processes can behave differently from normal local execution.
- UI and Audio Visualizer Overlap
I also faced a frontend design problem.
The dashboard was overlapping with the LiveKit audio visualizer.
I redesigned the dashboard and moved it into a vertical top-left layout.
This gave the visualizer and conversation area enough space.
What I learned
A good AI product is not only about the backend. The user interface also matters.
How to Run the Project
If you want to try the project yourself, you can follow these basic steps.
- Clone the Repository
First, clone the public GitHub repository:
git clone YOUR_GITHUB_REPOSITORY_URL
cd YOUR_PROJECT_FOLDER
- Install Dependencies
Install the required Python dependencies:
pip install -r requirements.txt
If the project has a separate frontend, go to the frontend directory and install its dependencies:
npm install
- Create the Environment File
Create a .env file in the appropriate project directory.
Add your own API keys and configuration values there.
For example:
MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
GOOGLE_API_KEY=your_key_here
LIVEKIT_URL=your_url_here
LIVEKIT_API_KEY=your_key_here
LIVEKIT_API_SECRET=your_secret_here
Never publish real API keys in your blog or GitHub repository.
Make sure .env is included in your .gitignore file:
.env
- Start the Agent
Run the agent using the command provided in the project repository.
For example:
python your_agent_file.py
Use the actual startup command from your repository because the command can be different depending on the project structure.
- Start the Frontend
If you are using the frontend dashboard:
npm run dev
Then open the local development URL shown in your terminal.
- Connect and Test
Once the agent and frontend are running:
Open the voice interface.
Connect to the LiveKit session.
Allow microphone access.
Start speaking with Sadie.
Test different features.
Ask a grammar question to test the Grammar Specialist.
Ask a maths question to test the Maths Specialist.
Ask a coding question to test the Full Stack Specialist.
You can also test memory, tools, and other features separately.
What I Learned
This challenge taught me that voice agents are much more than an LLM + TTS.
Here are some of my biggest lessons.
- Voice Agents Are More Than an LLM + TTS
A complete voice agent needs several systems working together:
Speech-to-Text + LLM + Text-to-Speech + Real-Time Transport + Tools + Memory
All these components need to work together smoothly.
- Latency Matters
In a normal chatbot, a small delay may not be a big problem.
In a voice conversation, users expect quick responses.
Even a small delay can make the conversation feel unnatural.
- Guardrails Matter
An agent needs clear instructions about what it can and cannot do.
Without proper guardrails, an AI agent may respond in unexpected ways.
- Good Tool and Handoff Architecture Matters
Tools and specialist agents make an AI system much more useful.
But they also need proper routing and context management.
A specialist should receive enough information to understand the conversation.
- Indian and Code-Mixed Conversations Need Proper Handling
Indian users may naturally switch between Hindi and English during a conversation.
For example:
"Mujhe Python ka ek function samjha do."
Handling this type of conversation properly was an important part of my project.
Future Improvements
This project is a starting point, and there are many things I would like to improve.
Better Multilingual Support
I want to add better support for more Indian languages and make conversations more natural across different languages.
More Specialist Agents
I would like to add more specialists for different domains.
For example:
Science
Career guidance
Interview preparation
Programming languages
General education
Better Analytics
The analytics dashboard can be improved with more detailed information such as:
Call duration
Response latency
Tool usage
Specialist handoffs
Call success rate
User activity
Lower Latency
Reducing the time between the user's speech and the agent's response would make conversations feel more natural.
Production Telephony
I would like to make the phone calling system more production-ready and reliable.
Better Personalization
The next version could use memory more effectively to personalize conversations for returning users while still respecting user permissions.
Architecture Diagram
The architecture of the project can be represented as:
┌───────────────┐
│ User │
└───────┬───────┘
│
Voice
↓
┌───────────────┐
│ LiveKit │
└───────┬───────┘
│
↓
┌───────────────┐
│ Deepgram │
│ STT │
└───────┬───────┘
│
↓
┌───────────────┐
│ Google Gemini │
│ LLM │
└───────┬───────┘
│
┌────────────────┼────────────────┐
↓ ↓ ↓
┌─────────┐ ┌─────────┐ ┌────────────┐
│ Memory │ │ Tools │ │ Specialists│
└─────────┘ └─────────┘ └────────────┘
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Grammar Maths Full Stack
Specialist Specialist Specialist
│ │ │
└─────────────┼─────────────┘
↓
┌─────────────────┐
│ Murf Falcon │
│ TTS │
└────────┬────────┘
│
↓
User
Final Thoughts
When I started this challenge, I thought building a voice agent mainly meant connecting speech recognition, an LLM, and text-to-speech.
After 10 days, I understand that a real voice agent is much more than that.
I worked with:
Voice
Memory
Tools
Telephony
Human escalation
Analytics
Multi-agent systems
Real-time communication
I also faced bugs, configuration problems, import errors, UI issues, and unexpected behavior.
But every problem taught me something new.
The biggest thing I learned is:
Building AI is not only about making it intelligent. It is also about making it useful, reliable, safe, and easy for people to use.
I am happy to complete 10 Days of AI Voice Agents — #VoiceForBharat Edition.
A big thank you to everyone behind the challenge for giving me the opportunity to build, experiment, debug, and learn.
Links
GitHub: https://github.com/ayushshinde13/10-days-of-AI-voice-agents-voice-for-bharat-challenge-2026
Demo Video: https://drive.google.com/file/d/1pTORvoqjRN6ZN0FbrkltpBRxejTVy4bi/view?usp=sharing
Challenge: 10 Days of AI Voice Agents — #VoiceForBharat Edition
Tech Stack
Python
LiveKit
Murf Falcon
Deepgram
Google Gemini
Next.js
Chart.js
SQLite
LiveKit SIP
Linphone
Discord Webhooks
Free Dictionary API
