My 10-Day Journey with Murf AI | VoiceForBharat Edition
Iโm excited to share my experience completing the 10 Days of Voice Agents โ VoiceForBharat Edition challenge by Murf AI.
Over these 10 days, I built and progressively enhanced an AI-powered interaction system using Murf Falcon, the fastest TTS API, along with LiveKit and Python.
The project started with basic real-time interaction and gradually evolved into a system capable of handling memory, tools, outbound communication, human escalation, call analytics, and specialist-agent handoffs.
1. The Problem and the Users
For this project, I selected a Financial Services use case.
Financial services often involve users asking questions, checking eligibility, requesting information, and needing assistance with different processes.
The goal was to explore how an AI-powered interaction system could make these experiences more natural and accessible.
Instead of requiring users to navigate complex interfaces or type every question, a voice-based interaction allows them to communicate naturally.
The AI system is designed to assist users while also understanding its limitations and knowing when to involve a human or another specialised agent.
2. Introducing Falcon AI
Falcon AI โ Intelligent Human Interaction & Agentic Automation System
Falcon AI is an AI-powered interaction system designed to demonstrate how intelligent agents can:
Communicate naturally with humans
Remember relevant information
Use tools to perform actions
Initiate outbound interactions
Escalate situations to humans
Analyse conversations
Hand off conversations to specialised agents
The project uses Murf Falcon for voice generation and LiveKit for real-time communication.
3. My 10-Day Journey
Day 1 โ Build the Foundation
The journey started by setting up the core real-time AI interaction system.
The goal was to establish the foundation required for users to communicate with the AI.
๐ฅ Day 1 Demo:
https://youtu.be/09HhFW0SQXI
Day 2 โ Personality, Role & Limits
The AI agent was given a defined personality, role, objectives, responsibilities, and safety boundaries.
The Financial Services scenario helped establish clear behaviour and limitations for the agent.
๐ฅ Day 2 Demo:
https://youtu.be/My_Wc8r5AdQ
Day 3 โ Personalised Frontend
The next stage focused on the frontend experience.
I created a personalised interface that allows users to interact with the AI system and observe the current state of the interaction.
๐ฅ Day 3 Demo:
https://youtu.be/RmWX0Qc_YYo
Day 4 โ Persistent Memory
The agent was given persistent memory so that relevant user information could be retained.
This allows returning users to receive more personalised and contextual interactions.
๐ฅ Day 4 Demo:
https://youtu.be/P-_4OXs09CU
Day 5 โ AI Tools & Actions
The agent was connected to tools that allow it to perform useful actions rather than simply generating responses.
For example, the Financial Services use case included functionality related to checking scheme eligibility.
๐ฅ Day 5 Demo:
https://youtu.be/hbNWKFElsI0
Day 6 โ Outbound Calls
The system was extended to support outbound interactions.
Instead of only waiting for users to initiate a conversation, the AI could proactively communicate with users through an outbound calling workflow.
๐ฅ Day 6 Demo:
https://youtu.be/tVoqUXaY5f0
Day 7 โ Human-in-the-Loop
One of the most important capabilities was human escalation.
The AI was designed to identify situations where it should involve a human instead of trying to handle everything independently.
This introduced a practical Human-in-the-Loop workflow.
๐ฅ Day 7 Demo:
https://youtu.be/cOmmg8e3gic
Day 8 โ Call Analytics Dashboard
Once the system could handle interactions, the next step was understanding those interactions.
I built a call analytics dashboard to monitor activity and gain insights into conversations and outcomes.
๐ฅ Day 8 Demo:
https://youtu.be/Hbqq_rakyHw
Day 9 โ Specialist Agent Handoff
The final major capability was specialist-agent handoff.
Instead of expecting one general-purpose agent to handle every situation, the system can transfer a conversation to a specialised agent when another agent is better suited to handle the request.
This introduced the concept of multi-agent collaboration.
๐ฅ Day 9 Demo:
https://youtu.be/5hWJMglt4Mg
4. How the System Works
The basic architecture can be represented as:
User โ Speech-to-Text โ AI Agent / LLM โ Memory & Tools โ Murf Falcon TTS โ User
Additional workflows extend the system:
Outbound Calls โ Human Escalation โ Analytics โ Specialist Agent Handoff
Main Components
Speech-to-Text
Converts the user's spoken input into text that the AI system can understand.
LLM / AI Agent
Processes the user's request, maintains conversation context, determines what action is required, and generates a response.
Murf Falcon
Murf Falcon provides the text-to-speech layer, converting AI-generated responses into spoken output.
LiveKit
LiveKit provides the real-time communication infrastructure required for the interaction.
Memory
Stores relevant information so returning users can receive more contextual responses.
Tools
Allow the AI agent to perform specific actions and interact with external functionality.
Human Escalation
Allows the AI to involve a human when a request falls outside its capabilities or requires human assistance.
Specialist Agents
Allow specific tasks to be transferred to agents designed for particular responsibilities.
5. The Difficult Parts
One of the most valuable parts of this challenge was discovering that not everything works perfectly on the first attempt.
Frontend Build Error
While running the frontend locally on localhost, I encountered a build error.
This required troubleshooting the development environment, checking configuration and dependencies, and identifying the issue before continuing with the project.
I have included screenshots/proof of the frontend build error as part of my project documentation.
This was important to me because real-world development is not only about showing the final working application. It is also about understanding and solving the problems encountered during development.
Other Challenges
I also faced difficulties with:
LiveKit configuration
Frontend and backend integration
Environment setup
API configuration
Outbound calling
Connecting multiple components
Runtime debugging
Testing the complete workflow
These challenges helped me improve my troubleshooting and development skills.
*6. What I Learned From the Challenges
*
When something failed, I learned to:
Read the complete error message.
Identify which component caused the problem.
Check dependencies and configuration.
Verify environment variables.
Test individual components separately.
Debug integrations step by step.
Test the complete workflow again.
These practical debugging experiences were one of the most valuable parts of the challenge.
7. Security
When publishing an AI project publicly, protecting sensitive information is extremely important.
I made sure not to expose:
API keys
Secret credentials
Phone numbers
Caller information
Private user data
Sensitive environment variables
API keys should always be stored securely using environment variables or an appropriate secret-management solution.
8. Source Code
The complete project source code is available on GitHub:
๐ GitHub Repository:
https://github.com/krithikgokuls/ai-voice-agents-murf-ai
The repository contains the implementation and supporting code for the AI interaction system built using Murf AI and LiveKit.
9. What I Learned
The biggest lesson from this challenge is that an AI voice system is much more than a system that can talk.
A useful AI agent should be able to:
Listen โ Understand โ Remember โ Act โ Analyse โ Escalate โ Collaborate
Through this challenge, I gained practical experience with:
Voice AI
Conversational AI
Agentic AI
Persistent memory
Tool calling
Real-time communication
API integration
Telephony
Human-in-the-Loop systems
Call analytics
Multi-agent workflows
10. What I Would Improve Next
There are several areas I would like to explore in future versions:
Better multilingual and code-mixed conversations
More advanced memory management
Improved analytics
Additional specialised agents
Better error handling
Production-ready deployment
Improved authentication
Enhanced frontend experience
Better monitoring and logging
More advanced Human-in-the-Loop workflows
2. Final Thoughts
Completing the 10 Days of Voice Agents โ VoiceForBharat Edition challenge was a valuable hands-on learning experience.
The difficult parts were just as important as the successful parts.
The frontend build error, LiveKit configuration, integrations, and debugging challenges taught me that building AI applications requires continuous experimentation, troubleshooting, and problem-solving.
This journey helped me understand how AI agents can move beyond simple conversations and become systems that can interact, remember, act, analyse, escalate, and collaborate.
Iโm grateful to Murf AI for providing this opportunity to learn and build with Voice AI.
Project
Falcon AI โ Intelligent Human Interaction & Agentic Automation System
Technology Stack
Murf Falcon | LiveKit | Python | Conversational AI | Agentic AI | Persistent Memory | Tool Calling | Telephony | Call Analytics | Multi-Agent Systems @Murfai
