🎙️ Building EduBuddy: My 10-Day Journey from a Voice Agent to a Multi-Agent AI Learning Companion

Introduction

Ten days ago, I signed up for 10 Days of Voice Agents — VoiceForBharat Edition, a challenge by Murf AI, with a starter pipeline and a vague idea: build something that lets people learn by talking instead of typing. I didn't have a finished architecture in my head. I had a track — Learning & Literacy — and a daily prompt to add one new capability.

What came out the other end is EduBuddy, an AI voice and text learning companion that gives learners exercises, evaluates their spoken answers, remembers who they are, calls them for practice, knows when to bring in a human, and can hand a maths question off to a specialist agent built just for that.

I'm writing this partly to document the build for myself, and partly because I think the more useful story isn't "look what I shipped" — it's how a voice agent actually gets assembled, piece by piece, and where it breaks along the way. If you're evaluating this as a project, or thinking about building something similar, this should tell you exactly what's under the hood.

The Problem

Most learning tools are text-first: read the material, type the answer, read the feedback. That works well for a lot of people, but it puts a barrier in front of learners who are more comfortable speaking than typing, or who find a keyboard-and-screen interface slower to engage with than a conversation.

Speaking is a lower-friction way to interact for a lot of learners. If a learner can talk through a problem, get evaluated on what they said out loud, and get a spoken response back, the interaction feels closer to being tutored than to filling out a form. That's the gap EduBuddy is aimed at — not a claim about literacy rates or any specific population, just a bet that voice removes friction that text adds for some learners.

What Is EduBuddy?

EduBuddy is a voice-and-text learning companion built for anyone who wants to practice a subject through conversation rather than through a form. A learner can talk to it in the browser or receive an outbound practice call, work through an exercise, have their spoken answer evaluated, and pick up where they left off next time because the agent remembers them.

The difference from a plain chatbot is that EduBuddy doesn't just answer questions — it runs a structured interaction: it can fetch an exercise, score an answer, decide whether to escalate to a human, or route a maths-specific question to a specialist agent that continues the same conversation. It's closer to a small system of cooperating agents and tools than to a single prompt.

Architecture

Learner
   ↓
Frontend / SIP
   ↓
LiveKit
   ↓
Deepgram STT
   ↓
Google Gemini
   ↓
Tools / SQLite / Agent Handoff
   ↓
Murf Falcon TTS
   ↓
Learner

• Frontend / SIP — entry point for the learner, either a Next.js browser session or an inbound/outbound phone call.
• LiveKit — handles real-time audio transport between the learner and the agent.
• Deepgram (Nova-3) — converts the learner's speech into text, with language configuration for English and Hindi/code-mixed input.
• Google Gemini — the reasoning layer; decides whether to respond directly, call a tool, store memory, escalate, or hand off to the maths specialist.
• Tools / SQLite / Agent Handoff — the functional layer: exercise fetching, answer scoring, learner memory, escalation records, call analytics, and specialist routing all live here.
• Murf Falcon — converts the agent's text response back into speech, described in the challenge as the fastest TTS API available.

10-Day Build Journey

Core Features

Voice + Personality

Murf Falcon handles text-to-speech, and Day 2 was spent defining EduBuddy's personality, objectives, and safety guardrails before any real functionality existed — so every later feature had a consistent voice and boundaries to operate inside.

Multilingual / Code-Mixed Interaction

Deepgram's STT is configured for multi-language detection to handle English, Hindi, and code-mixed speech. On the response side, correct script handling mattered — Hindi output needed to render in Devanagari rather than being Romanized, which took explicit attention in the STT config, the TTS config, and the system prompt, not just the LLM.

Learner Memory

Learner information relevant to future sessions is stored in SQLite — not a full transcript log, just what's useful for continuity. This is what lets EduBuddy recognize a returning learner instead of starting cold every time.

Learning Tools

fetch_next_exercise and score_spoken_answer are the two tools that turn the interaction into an actual practice loop: the agent hands the learner an exercise, the learner answers out loud, and the tool evaluates the response instead of the LLM eyeballing it.

Outbound SIP Calling

Using LiveKit's SIP integration, EduBuddy can initiate a call rather than only responding to one. This is what makes a "daily practice call" possible — the agent reaches the learner instead of waiting for them to open the app.

Human Escalation

Covered in detail below — this is the feature I'd point to first if someone asked what "responsible" looks like in this project.

Call Analytics

A dashboard backed by real SQLite call records, not placeholder numbers, so I could actually see how sessions were going rather than assume.

Specialist Agent Handoff

The main agent recognizes maths-specific requests and hands the conversation to a dedicated Maths Practice Specialist agent, which continues without asking the learner to repeat themselves.

Human Escalation

This feature gets its own section because it's the clearest example of designing for the agent's limits rather than pretending it doesn't have any.

Escalation triggers on two conditions:

• The learner shows severe frustration or emotional distress related to the learning session.
• The learner explicitly asks for a teacher, or stays stuck on the same concept after repeated attempts.

Before anything is shared, the agent asks the learner for permission. What gets recorded is a short, human-readable summary — an escalation ID, the reason, relevant details, and a severity/status field — never sensitive information like OTPs, passwords, PINs, or account details. The learner gets a reference ID and a clear explanation of what happens next. The point isn't that the agent tries harder to solve the problem itself — it's that it knows when to stop trying and hand off cleanly.

Multi-Agent Handoff

Instead of one large prompt trying to be good at everything, EduBuddy's main agent delegates maths-specific conversations to a separate specialist:

Learner: "Can you help me solve 3x + 9 = 24?"
Main Agent: "I will connect you to our maths specialist."
Maths Specialist: [continues the conversation directly, without asking
                    the learner to repeat the problem]

The benefit isn't just cleaner prompts — it's that each agent can be tuned, tested, and extended independently. Adding a second or third specialist later doesn't mean rewriting the main agent's entire personality and instruction set; it means adding a new handoff target.

Call Analytics

The dashboard tracks total calls, successful calls, failed calls, and failure categories, all computed from real SQLite records rather than hardcoded placeholder data. For EduBuddy, a "successful" call is defined as one where the learner actively engages and completes at least one exercise or practice activity — a call that ends before that point counts as unsuccessful. Caller identifiers are masked in the dashboard to protect privacy. I'm not going to quote specific numbers here, since the point of this section is the mechanism, not a metric I'd be inventing.

Technical Challenges

What Broke and What I Learned

The pattern across all three: a bug in STT, TTS, session handling, or SIP config can present exactly like "the AI is behaving wrong," even when the LLM itself is doing its job correctly. Debugging a voice agent means checking the whole pipeline, not just the prompt.

How to Build Your Own Voice Agent

1. Set up a LiveKit project for real-time audio transport.
2. Configure environment variables (see below) — never hardcode secrets.
3. Connect Deepgram for speech-to-text.
4. Connect Google Gemini as the reasoning layer.
5. Connect Murf Falcon for text-to-speech.
6. Get a basic STT → LLM → TTS conversation loop working end to end.
7. Add function tools so the agent can do things, not just talk.
8. Add SQLite for learner/session memory.
9. Add a human escalation path with clear triggers.
10. Add analytics from real usage data.
11. Add specialist agent handoff for domain-specific tasks.
12. Test through the browser first, then through SIP if you need telephony.

Practical Example

Here's the conceptual shape of the STT → LLM → TTS loop, written as pseudocode rather than a copy of any specific SDK call:

# Pseudocode — illustrates the flow, not a literal API
def handle_turn(audio_chunk):
    text = stt.transcribe(audio_chunk)          # Deepgram
    response = llm.generate(text, tools=TOOLS)  # Gemini + function tools

    if response.tool_call == "fetch_next_exercise":
        exercise = db.get_next_exercise(learner_id)
        response = llm.generate(exercise, context=text)

    audio = tts.synthesize(response.text)        # Murf Falcon
    return audio

Repository

GitHub: https://github.com/deepbisen-06/Voice-Agents

The repo contains the agent logic, the tool implementations, and the frontend used to run and test EduBuddy locally. Real credentials and caller data are excluded — you'll need your own API keys to run it.

What I Would Improve Next

• More specialist agents beyond maths
• Stronger multilingual and code-mixed support
• More reliable specialist handoff (fixing the silence issue for good)
• Better telephony error recovery
• More personalized learning paths per learner
• Deeper, more actionable analytics
• Better interruption handling mid-conversation

Key Lessons

1. A voice agent is a system, not a model — STT, LLM, TTS, transport, and session state all have to work together.
2. Language correctness (script, voice, config) has to be checked across the whole pipeline, not patched at the prompt level.
3. Giving an agent a way to say "I don't know, let me get a human" is a feature, not a limitation.
4. Specialist sub-agents scale better than one prompt trying to do everything.
5. Real data — for memory, analytics, or escalation — is worth the extra setup over hardcoded placeholders from day one.

Conclusion

I started this challenge with a pipeline that could just about hold a conversation. Ten days later, EduBuddy remembers learners, runs practice sessions, makes outbound calls, knows when to ask for human help, tracks its own performance, and delegates maths questions to a specialist built for the job. None of that happened in one step — it was ten small, deliberate additions, each one breaking something the previous day's work didn't expose.

Built during 10 Days of Voice Agents — VoiceForBharat Edition, using Murf Falcon.