Building LearnBuddy: A Voice-Powered Learning Agent with LiveKit and Murf Falcon

My 10-Day Voice AI Journey

Over the past 10 days, I worked on building a real-time voice AI agent as part of 10 Days of Voice Agents — VoiceForBharat Edition.

The project I built is called LearnBuddy, a voice-powered learning assistant designed to help students learn, practice technical concepts, and interact with an AI tutor through natural voice conversations.

What started as a basic voice assistant gradually became a more complete learning agent with personality, safety guardrails, memory, tools, human escalation, outbound calling, call analytics, and specialist-agent handoffs.

This journey taught me that building a useful voice agent is not only about making an AI speak. It is about designing how the agent understands users, responds safely, remembers context, uses tools, and knows when it needs help from another person or specialist.

What Problem Does LearnBuddy Solve?

Students often use text-based AI tools for learning, but typing every question is not always the most natural way to learn.

LearnBuddy is designed to make learning more conversational.

Instead of opening a chatbot and typing questions, a learner can simply speak to the agent and have a real-time conversation.

The main goal is to make technical learning more interactive and accessible through voice.

I focused the project on the Computer Engineering learning track.

LearnBuddy can help learners with topics such as:

Programming
Data Structures
Algorithms
DBMS
Operating Systems
Computer Networks
Computer Architecture
Microprocessors
Web Development

Voice is especially useful for this type of learning because students can ask questions naturally, practice concepts conversationally, and receive immediate spoken responses.

Why I Chose Voice

Text-based AI is powerful, but voice changes the experience.

With a voice agent, the learner does not need to continuously type questions and read long answers.

The interaction becomes closer to speaking with a tutor.

For example, a learner can ask for an explanation, request a practice problem, discuss the answer, and ask follow-up questions without leaving the conversation.

This makes the experience feel more natural and interactive.

The real-time communication layer was built using LiveKit Agents, which allows an AI agent to participate in realtime rooms and communicate with users through realtime media and data.

How LearnBuddy Works

At a high level, LearnBuddy follows a voice pipeline:

The main components are:

Speech-to-Text

The learner's voice needs to be converted into text so that the AI can understand the request.

I used Deepgram for speech recognition.

Large Language Model

The LLM processes the learner's request and decides what the agent should say or do.

For my implementation, I integrated a Google Gemini-based model.

Text-to-Speech

After generating a response, the text needs to be converted back into natural speech.

I used Murf for the voice output.

Murf's Falcon model is designed specifically for real-time voice applications, with low model latency and streaming audio capabilities. Murf describes Falcon as having model inference under 55 ms and time to first audio under 130 ms.

Realtime Transport

LiveKit connects the user and the voice agent through realtime communication.

LiveKit Agents supports voice pipelines using separate STT, LLM, and TTS components and also supports realtime telephony through SIP.

The Personality of LearnBuddy

One of the first important things I learned was that a voice agent should not simply be given an API and told to answer questions.

It needs a clear identity.

I created LearnBuddy with a learning-focused personality.

The agent is designed to:

Be friendly and encouraging
Explain concepts in simple language
Help learners understand mistakes
Encourage learners instead of simply giving answers
Stay focused on learning
Follow safety and conversation guardrails

This helped make the interaction feel more like a learning companion rather than a generic chatbot.

Guardrails and Safe Conversations

Another important part of the project was defining what LearnBuddy should and should not do.

The agent follows guardrails that keep the conversation focused on learning.

For example, when a learner asks for help, LearnBuddy should try to guide them instead of blindly producing an answer.

The agent also needs to handle conversations that are outside its intended purpose appropriately.

This taught me an important lesson:

A good AI agent needs boundaries, not just intelligence.

The quality of a voice agent depends not only on the model but also on the instructions and rules surrounding it.

Memory for Returning Learners

I also added memory to LearnBuddy.

The purpose of memory is to allow the system to remember useful information about a learner and provide a more personalized experience during future interactions.

Instead of treating every conversation as completely independent, memory can help the agent understand previous context.

For example, a learning assistant could remember a learner's preferred topics or previous learning activity.

Memory also introduced an important design consideration: user information should not simply be stored without thinking about privacy and consent.

This made me realize that personalization and responsible data handling need to be designed together.

Computer Engineering Practice Tool

One of the most useful features I implemented was a custom practice tool.

I created a function called:

get_coding_exercise

The tool can generate computer engineering practice exercises based on the learner's requested topic and difficulty level.

Supported areas include:

Python
Java
C++
Data Structures
Algorithms
DBMS
Operating Systems
Computer Networks
Computer Architecture
Microprocessors
Web Development

The tool supports different difficulty levels such as:

Beginner
Intermediate
Advanced

The important idea here is that the LLM does not have to do everything itself.

The agent can call a tool when a task requires a specific piece of functionality.

LiveKit Agents supports tool use as part of its agent framework, making it possible to connect agent reasoning with custom application logic.

Human Escalation

An important lesson from building an AI agent is that the AI should not always try to solve everything by itself.

Sometimes a learner may need help from a real person.

I therefore added a human escalation capability.

When the conversation reaches a situation where human assistance is appropriate, the agent can request human help instead of continuing indefinitely.

This makes the system more realistic because real-world AI systems should have a clear path to human assistance.

Outbound Calling

Another major feature I explored was outbound calling.

The goal was to allow the voice agent to communicate beyond the browser and interact through phone-based communication.

LiveKit provides telephony capabilities through SIP, which makes this type of integration possible.

This was one of the more challenging parts of the project because phone and SIP integrations require additional configuration compared with a normal browser-based voice conversation.

Call Analytics

I also worked on a call analytics dashboard.

The dashboard helps visualize information about voice interactions and call outcomes.

This is important because simply building a voice agent is not enough.

Once an agent is being used, we need to understand how it performs.

Analytics can help answer questions such as:

How many calls happened?
What were the outcomes?
How long were conversations?
How often did users need human help?
What types of interactions occurred?

This turns the project from just a voice demo into something closer to a complete application.

Specialist Agent Handoff

Another advanced capability I added was agent handoff.

Instead of having one agent handle every possible task, different agents can be designed for different responsibilities.

For example:

When a conversation requires a different capability, the current agent can transfer the conversation to another specialist.

LiveKit Agents provides support for multi-agent workflows and handoffs, allowing agents with different roles, tools, or reasoning behavior to work together.

This was one of the features that made me think beyond the idea of a single chatbot.

Challenges I Faced

The project did not work perfectly from the beginning.

In fact, some of the most useful lessons came from debugging.

1. API Quota Error

At one point, my Gemini integration returned a 429 Resource Exhausted error.

Initially, it looked like something was wrong with the agent.

After checking the error carefully, I found that the problem was related to the API quota rather than the basic voice-agent logic.

The lesson was simple:

Always read the actual API error before changing your code.

Not every runtime error is caused by your application logic.

2. SIP Calling Error

While working on outbound calling, I encountered an error similar to:

SipCallTo should be a phone number or SIP user,
not a full SIP URI

The problem was related to the destination format being provided to the SIP call.

This taught me that telephony integrations require careful attention to the exact format expected by the service.

It also showed me that adding phone capabilities is significantly more complex than simply adding another API call.

3. Frontend and Backend Communication

I also faced frontend errors such as:

Cannot read properties of undefined

and:

Failed to fetch

These errors forced me to check the connection between the frontend and backend rather than focusing only on the AI agent.

A voice AI application is a complete system.

The frontend, backend, realtime transport, APIs, models, and environment configuration all need to work together.

How You Can Build Your Own Voice Agent

If you want to build a similar project, the basic architecture is actually quite understandable.

You need four main AI components:

Then you need a realtime transport layer such as LiveKit to move the audio between the user and your agent.

LiveKit's official Voice AI Quickstart provides a ready-to-run starting point for building voice agents with Python or Node.js.

Step 1: Clone the Project

My complete source code is available here:

GitHub: https://github.com/akshata-patel30/murf-livekit-starter

Clone it using:

git clone https://github.com/akshata-patel30/murf-livekit-starter

Then enter the project directory:

cd murf-livekit-starter

Step 2: Configure API Keys

Never hard-code API keys directly into your source code.

Create an environment file such as:

.env.local

or the environment file expected by your project.

Example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key

Replace the placeholder values with your own credentials.

Do not publish these values on GitHub.

Murf's API documentation also recommends storing the API key securely as an environment variable rather than exposing it in source code.

Also make sure your .gitignore contains environment files such as:

.env
.env.local

Step 3: Install Dependencies

For a Python LiveKit agent, the current LiveKit quickstart uses Python and uv for dependency management.

After installing the required dependencies, start the backend development server.

For my project, I used:

uv run python src/agent.py dev

Keep this terminal running.

Step 4: Start the Frontend

Open another terminal and move into the frontend directory:

cd frontend

Install the frontend dependencies if required:

npm install

Then start the development server:

npm run dev

The frontend should then be available through the local development URL shown by Next.js.

Step 5: Connect and Test

Open the frontend in your browser.

Allow microphone access when the browser asks for permission.

Start a conversation with LearnBuddy.

Try different types of interactions:

Ask a normal learning question.
Ask for a practice exercise.
Test a different difficulty level.
Continue the conversation with follow-up questions.
Test memory and personalization.
Test human escalation.
Check the analytics dashboard.

LiveKit's documentation also provides a browser-based Agent Console for starting sessions and interacting with agents in realtime.

Keeping API Keys Safe

This is extremely important if you publish your project.

Never commit:

API keys
API secrets
Phone numbers
Private user information
Caller information
Database credentials
Private tokens

If you accidentally expose an API key, revoke or rotate it immediately.

Your public GitHub repository should contain the code and configuration examples, not your private credentials.

What I Learned

The biggest lesson from these 10 days was that building a voice agent is much more than connecting an LLM to a microphone.

I learned how different technologies work together:

Speech recognition
Large language models
Text-to-speech
Realtime communication
Frontend development
Backend development
Agent tools
Memory
Telephony
Human escalation
Analytics
Multi-agent workflows

I also learned that debugging is a major part of building AI applications.

Some problems were caused by code.

Others were caused by API limits, configuration, network communication, or external services.

Understanding the difference was one of the most valuable parts of the experience.

What I Would Improve Next

There are still many things I would like to improve in LearnBuddy.

Some of my future ideas include:

Better learner progress tracking
More personalized learning paths
More computer engineering topics
Improved analytics
Better multilingual support
More specialist agents
More robust phone integration
Production deployment
Better testing and evaluation
Improved response latency
More detailed learning reports

The goal would be to move LearnBuddy from a challenge project toward a more complete AI learning platform.

Final Thoughts

Building LearnBuddy over these 10 days completely changed the way I think about voice AI.

At the beginning, I thought a voice agent was mainly about making an AI talk.

By the end, I understood that a useful voice agent needs much more:

It needs a purpose, personality, memory, tools, safety rules, realtime communication, analytics, and a clear way to involve humans when necessary.

The combination of LiveKit for realtime communication and Murf Falcon for fast, natural text-to-speech made it possible for me to build a much more interactive learning experience.

This project was challenging, but every error and debugging session taught me something new.

I'm happy to have completed:

10 Days of Voice Agents — VoiceForBharat Edition.

And this is only the beginning of my journey into building realtime AI applications.

Project Links

GitHub Repository: https://github.com/akshata-patel30/murf-livekit-starter

LinkedIn: https://www.linkedin.com/in/akshata-patel-160886283

Technologies Used