Over the last 10 days, I built a voice agent as part of 10 Days of Voice Agents โ VoiceForBharat Edition.
What started as a simple voice conversation gradually became a much more complete system โ one that can remember users, use tools, make outbound calls, escalate to a human, track call outcomes, and hand conversations over to a specialist agent.
The project was built using LiveKit and Murf Falcon, with the goal of making government-scheme and financial assistance information easier to access through natural voice conversations.
1. The Problem I Wanted to Solve
Government schemes can be extremely useful, but discovering the right scheme, understanding eligibility, and figuring out what to do next can be confusing.
For many users, especially those who are more comfortable speaking than typing, a voice interface can make this interaction much simpler.
Instead of navigating through multiple pages and forms, a user can simply talk to an assistant and ask questions such as:
- What government schemes are available for me?
- Am I eligible for a particular scheme?
- What information do I need?
- Can someone help me with this?
This became the core idea behind my VoiceForBharat project: use voice as a more natural interface for accessing useful government and financial information.
2. What I Built
The final agent is more than a basic question-and-answer bot.
It can:
- Have natural voice conversations
- Use an Indian voice powered by Murf Falcon
- Handle Indian languages and code-mixed conversations
- Follow a defined personality and safety guardrails
- Remember information about returning users
- Use tools to retrieve or calculate useful information
- Make outbound phone calls
- Escalate conversations when a human is needed
- Track call outcomes
- Hand conversations from the main agent to a specialist agent
The idea was to make the agent feel less like a demo chatbot and more like an actual voice assistant.
3. How the System Works
At a high level, the voice pipeline looks like this:
User โ Speech-to-Text โ LiveKit Agent โ LLM โ Tools / Memory โ Murf Falcon TTS โ User
The user speaks into the interface. Their speech is converted into text, processed by the agent and LLM, and the response is converted back into speech using Murf Falcon.
LiveKit handles the real-time voice communication, while the agent logic controls the conversation, tools, memory, escalation, and handoffs.
The system also has additional paths for things like human escalation, outbound calls, and specialist-agent handoffs.
4. Building the Agent Step by Step
Voice and Personality
The first step was getting the basic conversation working.
I defined what the agent should do, how it should communicate, and what it should avoid doing. This was important because a useful voice agent needs more than an LLM โ it needs clear instructions and boundaries.
I also wanted the experience to feel natural for Indian users, including conversations where users switch between English and Hindi.
Murf Falcon provided the voice layer, making the responses feel much more conversational than a purely text-based interaction.
Memory
One of the next important steps was giving the agent memory.
Instead of treating every conversation as completely new, the system can store relevant user information and use it when the user returns.
This introduced an important lesson: memory is not just about storing information; it is also about correctly identifying the user and retrieving the right information.
While testing this, I ran into issues where user identification and stored information did not behave exactly as expected. Debugging those issues helped me understand how important consistent user IDs and clean memory logic are in a conversational system.
Tools
The agent also uses tools instead of relying entirely on the LLM's internal knowledge.
This is important for practical applications because tools allow an agent to work with structured information and perform specific actions.
The general pattern became:
User asks โ Agent understands intent โ Appropriate tool is used โ Result is explained naturally through voice.
Outbound Calls
Another major part of the project was adding outbound calling.
This allowed the agent to move beyond a browser-based demo and interact through an actual phone-call workflow.
Getting this working required connecting the voice agent with the telephony flow and testing the complete call lifecycle.
It also introduced new considerations around caller information, user identification, and making sure private information never gets exposed publicly.
5. Human Escalation
A voice assistant should not try to solve everything itself.
One of the features I added was the ability to escalate a conversation when the user needs human assistance.
For example, if a user reaches a point where an automated response is not enough, the agent can move toward a human-support path instead of continuing to guess.
This was one of the most interesting parts of the project because it changes the role of the AI.
The goal isn't:
AI should answer everything.
The goal is:
AI should know when it can help and when it should involve a human.
6. Specialist Agent Handoff
I also implemented a specialist-agent workflow.
Instead of having one giant agent responsible for every possible task, the main agent can hand a conversation to a more specialized agent when appropriate.
This creates a more scalable architecture.
For example:
Main Agent โ Understands User โ Determines Need โ Specialist Agent โ Handles Specific Task
This is much closer to how a real conversational system could be structured as it grows.
7. Call Analytics
Another feature I added was call analytics.
The system records useful information about calls so that conversations can be analyzed after they happen.
This makes it possible to look beyond:
โDid the agent answer?โ
and start asking:
- Did the call succeed?
- What happened during the conversation?
- Was a human needed?
- Which path did the call take?
- What was the final outcome?
This was especially useful because it showed me that building a voice agent isn't only about real-time interaction. Observability matters just as much.
8. The Challenges
The project definitely did not work perfectly on the first attempt.
One of the biggest challenges was debugging the interaction between different parts of the system.
For example, while working on memory, I encountered situations where the agent appeared to work correctly but retrieved the wrong stored information because the user identity being used by different parts of the system wasn't consistent.
I also faced issues while setting up and testing outbound calling and the handoff flow.
These problems taught me an important lesson:
A voice agent is a system, not just a prompt.
When something goes wrong, the problem may be in the prompt, memory layer, tool, user identification, transport, telephony, or agent routing.
Breaking the system into smaller components and testing each path separately made debugging much easier.
I also learned the importance of testing more than the happy path.
A good test isn't only:
โAsk the agent a question and see if it responds.โ
It should also include:
โWhat happens when the user needs a human?โ
โWhat happens when the user returns?โ
โWhat happens when the agent needs a specialist?โ
โWhat happens when a call ends?โ
9. How You Can Build Your Own Voice Agent
If you want to build a similar system, the basic architecture doesn't need to be complicated at first.
You need four major components:
1. Speech-to-Text
Converts the user's voice into text so the agent can understand the conversation.
2. LLM
Processes the user's request and decides what response or action is appropriate.
3. Text-to-Speech
Converts the agent's response back into natural speech.
In my project, I used Murf Falcon for this layer.
4. Real-Time Transport
Handles the live audio connection between the user and the agent.
I used LiveKit for the real-time voice infrastructure.
Once the basic pipeline works, you can gradually add:
Memory โ Tools โ Telephony โ Human Escalation โ Analytics โ Specialist Agents
That is a much better approach than trying to build everything at once.
--> Escalations Dashboard-
10. Running the Project
The project is available here:
GitHub: https://github.com/ZikraRahman/murf-livekit-starter
The general setup process is:
- Clone the repository.
- Install the project dependencies.
- Create your local environment configuration.
- Add your API credentials to the local environment file.
- Start the backend/agent.
- Start the frontend if required.
- Connect to the voice agent.
- Test a normal conversation.
- Test the escalation and specialist-agent paths.
Never commit your .env file or API keys to GitHub.
Phone numbers, caller information, user identifiers, and other private data should also never be included in a public repository.
11. What I Would Improve Next
There is still a lot I would like to improve.
Some of the next things I would explore are:
- Better multilingual support
- More government-scheme integrations
- More robust memory and user identification
- Better call analytics
- Improved error handling
- More specialist agents
- Better monitoring and observability
- More extensive real-world testing
- Improved accessibility for users with limited digital literacy
The most exciting part is that the architecture can keep growing without requiring the entire system to be rebuilt from scratch.
12. What I Learned
The biggest thing I learned during these 10 days is that building a voice agent is very different from simply connecting an LLM to a microphone.
A useful voice agent needs:
Voice + Intelligence + Tools + Memory + Guardrails + Real-Time Communication + Observability
And, most importantly, it needs to be designed around the user.
I also learned that debugging is a major part of building these systems. Things like user IDs, tool responses, call states, handoffs, and telephony flows can all create unexpected problems.
But every failure made the architecture a little clearer.
What started as a simple conversational agent ended up becoming a much more complete voice system with memory, tools, outbound calling, human escalation, analytics, and specialist handoffs.
13. Final Links
GitHub: https://github.com/ZikraRahman/murf-livekit-starter
LinkedIn: https://www.linkedin.com/in/zikra-rahman-ab32263bb/
(All video demos are uploaded on LinkedIn.)
Final Thoughts
The 10 Days of Voice Agents โ VoiceForBharat Edition challenge gave me the opportunity to go from building a basic voice interaction to thinking about what a production-oriented voice system actually needs.
The biggest takeaway for me is simple:
Voice AI becomes much more powerful when it is connected to real actions, real workflows, and real users.
And this is only the beginning.
