From a Simple Voice Agent to a Complete AI Learning Support System
Over the last 10 days, I participated in 10 Days of Voice Agents – VoiceForBharat Edition, where the goal was to build an AI voice agent around a grassroots-level problem in India.
I chose the Learning & Literacy track and built an AI Learning Companion designed to help learners through voice-based interaction.
What started as a simple voice agent gradually became a complete learning support system with memory, practice tools, outbound calls, human escalation, analytics, and specialist-agent handoffs.
The interesting part wasn't having access to these technologies—everyone participating in the challenge had access to the same core resources. The challenge for me was figuring out how to connect them into one meaningful learner experience.
The Problem
Learning support is often designed around screens, typing, and formal communication.
For many learners, especially those who are more comfortable speaking than typing or communicating in a mixture of Indian languages and English, a conversational voice interface can make learning feel more natural.
My goal was therefore not to build another chatbot that simply answers questions.
I wanted to build an assistant that could support a learner throughout the learning process:
Understand → Explain → Practice → Remember → Follow up → Escalate → Measure
The agent is designed around the Learning & Literacy track and supports natural interaction across English, Hindi, and Hinglish.
*What I Built
*
The final AI Learning Companion brings together several capabilities into one system.
1. Voice-first learning
The learner can communicate naturally with the agent instead of relying only on text.
The voice pipeline uses Murf Falcon for fast text-to-speech, allowing the agent to respond naturally during conversations.
The frontend also provides clear states such as connecting, listening, speaking, and ending a call so the learner knows what the system is doing.
2. A defined personality and boundaries
The agent isn't designed to answer everything without limits.
It has a specific role as a learning companion, defined objectives, and guardrails for the Learning & Literacy use case.
For example, the agent should never shame a learner for giving a wrong answer and should not claim that a learner has a learning disability.
It also knows when a request is outside its role instead of pretending to know everything.
3. Consent-based memory
A learner shouldn't have to introduce themselves from scratch every time.
I added persistent memory so the agent can remember useful learning information such as:
- Learner name
- Current learning level
- Topics covered
- Areas where the learner struggles
- Language preference
The agent asks for permission before saving information.
When the learner returns, the agent can use that information to continue the learning experience instead of starting from zero.
*4. Learning tools instead of relying only on the LLM
*
One of the important steps was teaching the agent to use an actual function.
I created a "fetch_practice_question(topic, level)" tool.
Instead of placing every possible question directly into the prompt, the agent can call the tool when a learner requests practice.
The tool searches a hand-built local practice-question dataset covering subjects and difficulty levels.
For example:
«"I'm a beginner. Give me a science practice question."»
The agent can call the tool with the learner's topic and level, receive the result, and then speak the question.
The tool also has a strict failure path. If the requested topic or level isn't available, the agent does not invent a question and instead tells the learner that suitable content isn't currently available.
5. Outbound learning calls
Learning doesn't have to end when the learner closes the browser.
I also added outbound calling capabilities for the Learning & Literacy use case.
The idea is simple: a learner can receive a scheduled practice call and continue their learning through voice.
Because an outbound call is unexpected from the user's perspective, the agent clearly explains who is calling, why it is calling, and how the learner can stop the interaction.
6. Human teacher escalation
AI shouldn't try to solve every learning problem.
I added a human-help flow for situations where a learner is struggling or specifically needs help from a teacher.
The agent first explains that it can create a request for human assistance and asks for permission before sharing the relevant information.
If the learner agrees, the system creates a short escalation request containing useful context rather than sending the entire conversation.
The learner receives a reference ID and an honest explanation of what happens next.
This creates a simple AI + human support loop instead of treating the AI as a replacement for teachers.
7. Call analytics
I also added a simple analytics dashboard to measure whether conversations actually achieve their intended goal.
For the Learning & Literacy agent, I defined a successful call as a learner completing a practice exercise.
The dashboard tracks:
Total Calls | Successful Calls | Failed Calls
The values come from actual agent sessions rather than hardcoded numbers.
This provides a basic way to measure whether the voice agent is actually accomplishing its learning objective.
8. Specialist agent handoff
The main agent doesn't need to be an expert at everything.
For Day 9, I introduced a separate Maths Practice Specialist.
The main Learning Companion handles general learning conversations. When a learner needs focused mathematics practice, the main agent can announce the handoff and transfer the conversation to the specialist.
The specialist continues with the context of the conversation, so the learner doesn't have to explain the problem again.
This creates a simple multi-agent architecture:
Main Learning Agent → Maths Specialist
while keeping each agent's responsibility focused.
*The Technology Behind It
*
The voice agent combines the major components required for a real-time voice AI system:
Speech-to-Text
Converts the learner's spoken words into text that the AI agent can understand.
Large Language Model
The LLM handles conversation, decides what the learner needs, follows the agent's instructions, and decides when tools or specialist agents are required.
Tools and Functions
Functions allow the agent to interact with external logic and data instead of relying entirely on generated responses.
Examples include:
"fetch_practice_question"
and the memory and human-escalation functions.
Text-to-Speech
Murf Falcon converts the agent's responses into natural speech.
Real-Time Transport
The voice interaction is handled through the real-time voice-agent infrastructure, allowing the learner and agent to communicate conversationally.
The Hard Parts
The challenge definitely wasn't a straight line from idea to working product.
One of the earliest problems I faced was getting the voice agent to consistently respond during conversations. At one point, the agent could respond to the first interaction but fail to continue properly.
I tested different configurations and investigated whether the problem was related to the LLM and agent setup.
Another important lesson came while implementing tools.
It isn't enough to create a function and assume the AI will use it correctly. The tool description matters because the model uses that description to decide when the function should be called.
I also had to make sure that a failed lookup didn't turn into a hallucinated answer. For example, if a requested learning topic isn't available in the dataset, the agent should say that rather than inventing content and presenting it as if it came from the dataset.
The same principle became even more important when building human escalation. The agent needed to know when to stop, ask permission, create a useful summary, and give the learner a reference ID instead of pretending it could solve everything.
These problems taught me that building an AI agent isn't just about connecting APIs.
The difficult part is designing reliable behavior around the AI.
How You Can Build Your Own Voice Agent
A basic voice agent can be broken into a few core components:
User Speech
↓
Speech-to-Text
↓
LLM / Agent
↓
Tools / Memory / Business Logic
↓
Text-to-Speech
↓
User
For real-time interaction, you also need a transport layer that connects the user and the voice agent.
In my project, the implementation is based on the Murf LiveKit starter architecture, with the agent logic, tools, memory, and frontend built around that foundation.
*Getting Started
*
Clone the public repository:
git clone https://github.com/VaibhavAran/murf-livekit-starter.git
Then follow the setup instructions in the repository's "README.md" to install the required dependencies and start the backend and frontend.
API Keys
Keep API keys in environment variables.
For example:
.env
Never place API keys directly inside source code and never commit ".env" files containing real credentials to GitHub.
Running the Agent
After configuring the required environment variables, start the backend and frontend using the commands documented in the repository README.
Open the frontend, allow microphone access, connect to the agent, and start a conversation.
A good first test is simply:
«"Hello, can you help me learn something?"»
Then test one feature at a time—memory, practice tools, escalation, and specialist handoff.
*What I Learned
*
The biggest lesson from this challenge is that building a voice agent is much more than making an LLM speak.
A useful agent needs:
- A clear job
- Well-defined limits
- Reliable tools
- Memory with consent
- Human escalation
- Good user feedback
- Measurable outcomes
- Focused specialist capabilities
And perhaps most importantly, the agent needs to know when it should not act on its own.
Everyone participating in this challenge had access to many of the same resources. The interesting part was deciding how to use those building blocks to create a coherent experience.
For me, the goal was to connect every capability around one learner journey instead of treating each feature as an isolated demo.
*What I'd Build Next
*
There is still a lot I'd like to improve.
I'd like to expand the curriculum and support more Indian languages, make the learning experience more adaptive, improve low-bandwidth behavior, introduce more specialist agents, and build a stronger teacher dashboard.
I'd also like to measure actual learning progress over longer periods rather than focusing mainly on individual calls.
The long-term goal would be to move from an AI that simply helps a learner during a conversation to a system that can understand how that learner is progressing over time.
*Final Thoughts
*
The 10 Days of Voice Agents – VoiceForBharat Edition gave me a chance to build something from a simple voice interaction into a much more complete system.
The final AI Learning Companion can:
Talk → Understand → Remember → Practice → Use Tools → Follow Up → Escalate → Measure → Delegate
But the biggest takeaway isn't the number of features.
It's learning how to make AI behave responsibly and usefully in a real user journey.
I'm grateful to Murf AI for creating a challenge that pushed me to build, debug, experiment, and think beyond a basic chatbot.
Links
GitHub Repository: "https://github.com/VaibhavAran/murf-livekit-starter.git" (https://github.com/VaibhavAran/murf-livekit-starter.git)
Built as part of 10 Days of Voice Agents – VoiceForBharat Edition, powered by Murf Falcon.
