My journey through #VoiceForBharat — building a voice agent that talks, remembers, teaches, calls, escalates, measures itself, and knows when to bring in a specialist.

The Problem and the Users

Millions of Indian students want extra practice help — a Python question to try, a grammar concept to review, a maths problem to work through — but not everyone has access to a tutor who's available on demand, comfortable switching between English, Hindi, and Hinglish, and patient enough to explain the same thing three different ways.

I built Bharat Buddy, an AI Voice Tutor, to fill that gap. It's built for the Learning & Literacy track of the VoiceForBharat challenge. The core idea: a student should be able to just talk to something — no typing, no navigating menus — and get real practice, in whatever language mix feels natural to them.

Voice matters here specifically because a lot of the students this is meant for are more comfortable speaking than typing in English, and a spoken back-and-forth feels a lot closer to sitting with an actual tutor than a chat window does.

What Bharat Buddy Does

By the end of the challenge, Bharat Buddy can:

Hold a natural, real-time voice conversation in English, Hindi, or Hinglish
Remember returning students — their level, topics covered, common mistakes
Fetch real practice exercises (Python, Maths, English Grammar, Computer Science) instead of inventing them on the fly
Make outbound phone calls to reach a student directly
Recognize when a student is distressed or needs a real human teacher, ask permission, and escalate — with a reference ID
Track every call's outcome (successful / failed) on a live analytics dashboard
Hand off maths questions to a dedicated Maths Specialist agent mid-conversation, without making the student repeat themselves
How the System Works

At a high level, every voice agent — Bharat Buddy included — is four pieces wired together:

🎙️ Microphone audio
↓
Speech-to-Text (Deepgram)
↓
LLM (Groq — Llama 3.3)
↓
Text-to-Speech (Murf Falcon)
↓
🔊 Spoken response

All of this runs on LiveKit for real-time audio transport, which handles the actual streaming of audio in and out with low latency — the part that makes it feel like a conversation instead of a walkie-talkie.

The stack:

🎙️ Murf Falcon — voice generation (the fastest TTS API, and it shows in how quickly Bharat Buddy starts speaking)
⚡ LiveKit — real-time transport
🎧 Deepgram — speech-to-text
🤖 Groq (Llama 3.3) — the LLM powering conversation and tool use
🐍 Python — agent logic
🗄️ SQLite — memory and call analytics storage
📞 SIP + Linphone — outbound phone calls
The Most Important Features

1. A voice and personality built for India

Day 1 and 2 were about getting the basics right: real-time voice in, voice out, and a personality that actually sounds like a friendly Indian tutor rather than a generic assistant. Murf Falcon's speed made a real difference here — low latency between the student finishing a sentence and Bharat Buddy starting to reply keeps the conversation feeling alive instead of laggy.

Getting Hindi and Hinglish right took real tuning. It's not enough to translate — the agent needs to know that Hindi should always render in Devanagari script, never romanized, and that a student can flip between English and Hindi mid-sentence without the agent getting confused or switching tone awkwardly.

1. Memory and context

A tutor who forgets you every session isn't much of a tutor. Bharat Buddy stores each student's language preference, current level, topics covered, and common mistakes in SQLite, and carries conversation context within a session so follow-up questions make sense without the student re-explaining themselves.

1. A real learning tool, not invented answers

Early on, an LLM will happily invent a practice question if you ask for one — which is exactly what you don't want in an education product, because the "answer" and "explanation" might just be confidently wrong. So Bharat Buddy has a get_next_exercise tool that pulls from a real local dataset (Python, Maths, English Grammar, Computer Science, beginner/intermediate) — and the system prompt explicitly forbids the agent from making one up if the tool comes back empty.

1. Outbound phone calls

This was the point where Bharat Buddy stopped being "a thing you open in a browser" and became something that can actually reach a student. Using LiveKit's SIP integration with Linphone, the agent can dial out and start a real-time voice conversation over an actual phone call — introducing itself, explaining why it's calling, and giving the person an easy way to end the call if it's not a good time.

1. Knowing when to ask for human help

Not every problem should be solved by the AI. If a student sounds distressed — frustrated, hopeless, ready to give up — or explicitly asks for a real teacher, Bharat Buddy:

Responds with empathy first
Asks permission before sharing anything: "May I share a short summary with a human helper?"
Only escalates if the student agrees
Sends a clean, minimal summary (who, what happened, what was already tried, urgency, language, preferred follow-up) to a real Discord channel — never passwords, OTPs, or account details
Gives the student a reference ID and an honest next step

If the student says no, nothing gets sent. That permission step matters — it's the difference between a helpful escalation and a system that shares a kid's private conversation without asking.

1. A call analytics dashboard

You can't improve what you don't measure. I defined success simply for this track: a call is successful if the student receives and engages with at least one exercise. Every call's outcome gets logged to SQLite, and a small Flask dashboard shows three real numbers — Total Calls, Successful Calls, Failed Calls — pulled live from the database, not hardcoded.

1. Handing off to a specialist

The last piece: one agent shouldn't try to be an expert at everything. I built a dedicated Maths Practice Specialist — a separate agent with its own instructions, focused only on step-by-step maths help. When a student asks for maths practice, Bharat Buddy announces the handoff ("Let me connect you to our Maths Specialist"), passes the full conversation context along, and the specialist introduces itself and continues — the student never has to repeat their question.

Challenges and How I Overcame Them

Tool-calling reliability with a small model. I initially used llama-3.1-8b-instant for speed, but as soon as I added a third tool (the maths handoff), the model started occasionally writing out the function call as visible text instead of actually invoking it — something like literally speaking out loud instead of calling the tool. Switching to llama-3.3-70b-versatile fixed the reliability of tool calls significantly. Smaller, faster models are great for simple conversation, but multi-tool orchestration benefits from a stronger model.

Rate limits from a bloated system prompt. My system prompt grew organically over the days — by Day 9 it was over 500 lines. Every single request sends that whole prompt to the LLM, and on Groq's free tier (6,000 tokens/minute), a long prompt plus a few tool calls burned through the quota almost immediately. The fix was trimming the prompt down to its essential rules — same behavior, roughly a tenth of the length — which freed up a lot more headroom for actual conversation.

A silent shutdown crash. My call-tracking shutdown callback was defined as a regular def, but LiveKit expects an async def and tries to await it — which threw a TypeError every time a call ended and quietly broke the call-outcome logging. One-line fix once I found it in the logs, but a good reminder to actually read the full traceback instead of assuming the visible error is the only problem.

Help the Reader Build Their Own Voice Agent

If you want to build something like this, here's the minimum path:

The four core components:

Speech-to-text — converts the caller's voice into text (I used Deepgram)
An LLM — understands the text, decides what to say or which tool to call (I used Groq)
Text-to-speech — converts the reply back into voice (Murf Falcon — genuinely fast, which matters a lot for how natural the conversation feels)
Real-time transport — streams audio both ways with low latency (LiveKit)

Setting up and running the project:

bash

clone and enter the project

git clone
cd murf-livekit-starter/backend

install dependencies (this project uses uv)

uv sync

copy the example env file and fill in your keys

cp .env.example .env.local

run the agent

python src/agent.py dev

Then start the frontend separately:

bash
cd frontend
pnpm install
pnpm dev

Where to add API keys without exposing them: All secrets go in .env.local — LiveKit URL/key/secret, your TTS/STT/LLM provider keys, and any webhook URLs. This file is listed in .gitignore (as .env.*) so it never gets committed. Never hardcode a key directly in your source files.

Connecting and testing: Open the frontend at http://localhost:3000, allow microphone access, and start talking. Watch your agent's terminal logs alongside the conversation — that's where you'll see tool calls, errors, and exactly what the LLM decided to do.

My repository: (add your public GitHub repo link here)

What I'd Improve Next
Add more specialist agents (English speaking practice, exam prep) with proper routing logic
Track failure types (declined, incomplete, tool error, no response) instead of just success/failure
Add a "hand back to main agent" flow so the specialist can return control when its job is done
Move off the free LLM tier for anything approaching production use — rate limits are a real constraint
Closing

Ten days ago, Bharat Buddy was a script that could turn text into speech. Now it's a system that listens, remembers, teaches, calls, knows its limits, measures itself, and delegates. The most useful lesson from the whole challenge: a good voice agent isn't defined by how much one model can do — it's defined by how well you design the boundaries around what it shouldn't try to do alone.

Built as part of 10 Days of Voice Agents — VoiceForBharat Edition, using Murf Falcon, the fastest TTS API.

VoiceForBharat #10DaysOfVoiceAgents #MurfAI #MurfFalcon #VoiceAI #LiveKit #Python #ConversationalAI