From a Simple Voice Conversation to a Multi-Capability Healthcare Voice Agent
Over the past 10 days, I had the opportunity to participate in 10 Days of Voice Agents — VoiceForBharat Edition, a challenge focused on learning how to build practical, real-world voice AI agents.
Instead of treating the challenge as just a series of coding tasks, I wanted to build something around a problem that genuinely matters: making healthcare access more conversational and accessible through voice.
That idea became AarogyaMitra — a voice-first healthcare access assistant designed to interact with users naturally, provide useful assistance, use tools when required, remember relevant user context, and involve humans or specialist agents when the situation requires it.
This article documents my journey, the architecture behind the project, the important features I built, the challenges I faced, and what I learned while developing a real-time voice AI system.
What is AarogyaMitra?
AarogyaMitra is a voice AI assistant focused on the Health Access track of the VoiceForBharat challenge.
The goal is simple:
Make healthcare assistance more accessible through natural voice conversations.
Many digital healthcare experiences assume that users are comfortable reading, typing, navigating menus, and interacting with conventional applications.
Voice can provide a more natural alternative.
Instead of searching through menus or typing a question, a user can simply speak to the assistant and have a conversation.
AarogyaMitra is designed around this idea.
The core objectives are:
- Make healthcare-related interactions more conversational
- Provide a simple voice-first interface
- Use AI tools when additional information or actions are required
- Maintain useful context during conversations
- Follow safety-oriented guardrails
- Escalate situations that require human assistance
- Route specialized requests to a specialist agent
AarogyaMitra is intended to assist users, not replace qualified healthcare professionals. Safety and appropriate escalation are therefore important parts of the system.
Why Voice AI for Healthcare Access?
Voice changes the way users interact with technology.
A traditional application might require:
Open app → Find feature → Type query → Read result
A voice assistant can reduce that interaction to:
Speak → Listen → Respond
This can be particularly useful when users prefer speaking over typing or when navigating a conventional interface is inconvenient.
For healthcare access, conversational interaction can also make the experience feel more approachable.
However, healthcare is a sensitive domain. A voice agent should not simply answer everything confidently.
That is why AarogyaMitra was designed with guardrails, escalation paths, and specialist handoffs rather than treating the AI as an unrestricted chatbot.
What I Built During the 10-Day Challenge
The biggest lesson from this challenge was that a useful voice agent is much more than speech input and speech output.
Over the 10 days, I gradually added capabilities that transformed the initial agent into a more complete voice AI system.
1. Natural Voice Conversations
The foundation was creating an agent capable of having a real-time conversation with the user.
The basic interaction follows:
User speaks
↓
Speech-to-Text
↓
AI Agent / LLM
↓
Response Generation
↓
Text-to-Speech
↓
User hears the response
This creates a conversational loop rather than a traditional request-response interface.
2. Indian Voice with Murf Falcon
One of the important parts of the project was giving the agent a natural voice.
For text-to-speech, I used Murf Falcon, the fastest TTS API highlighted by the challenge.
The goal was not simply to convert text into audio, but to make the interaction feel more natural and suitable for a voice-first application.
The voice layer is particularly important because even a technically powerful AI system can feel frustrating if responses are slow, unnatural, or difficult to understand.
3. Personality and Safety Guardrails
An AI agent needs more than a model.
It needs clear instructions defining:
- Its role
- What it should help with
- How it should communicate
- What it should avoid
- When it should ask for clarification
- When it should involve a human
For a healthcare-oriented agent, these boundaries are especially important.
AarogyaMitra is designed to provide assistance and information while avoiding the assumption that an AI system should independently make high-stakes medical decisions.
This was one of the important lessons from the challenge:
A good AI agent is not only defined by what it can do, but also by what it knows it should not do.
4. Memory for Better Conversations
Another major step was adding memory.
Without memory, every interaction can feel disconnected.
With memory, the system can maintain relevant information about returning users and provide a more continuous experience.
The concept can be represented as:
Returning User
↓
Retrieve Relevant Context
↓
Voice Conversation
↓
Agent Uses Context
↓
More Personalized Interaction
Memory makes the agent feel less like a collection of isolated questions and more like an ongoing assistant.
At the same time, memory must be handled carefully, especially for sensitive information.
5. Tools: Giving the Agent the Ability to Do More
A language model alone is limited to generating responses.
Tools allow an agent to interact with external functionality.
Instead of asking the model to invent an answer, the agent can use a tool when the required information or action belongs outside the model itself.
Conceptually:
User Request
↓
Agent Understands Intent
↓
Does this require a tool?
↓
Yes → Execute Tool
↓
Process Result
↓
Generate Voice Response
This was an important transition in my project because the agent became capable of doing more than simply having a conversation.
6. Human Escalation
One of the most important lessons I learned was that AI should not always try to handle everything itself.
Some situations require human assistance.
AarogyaMitra therefore includes a path for escalating conversations when the AI should not continue independently.
The basic flow is:
User
↓
AI Agent
↓
Determine whether human help is required
↓
Human Escalation
↓
Specialist / Human Support
This is particularly important in sensitive domains such as healthcare.
A strong AI system should know when to say:
"This is something where human assistance would be more appropriate."
7. Outbound Voice Calls
The challenge also introduced outbound calling.
This expanded the project beyond a user simply opening a webpage and talking to an agent.
The system can be used as part of an outbound voice workflow, creating another way for an AI agent to interact with users.
This introduced a completely different set of considerations around:
- Call initiation
- Voice interaction
- User experience
- Call outcomes
- Reliability
- Privacy
Building this made me realize that voice AI becomes significantly more complex when it moves from a controlled browser environment into real-world communication workflows.
8. Call Outcome Tracking and Analytics
A production-oriented voice system should not only make calls.
It should also help us understand what happened during those calls.
Tracking outcomes provides useful information about:
- Successful interactions
- Failed calls
- Escalations
- Conversation outcomes
- Overall system behavior
This creates a feedback loop:
Voice Interaction
↓
Call Outcome
↓
Analytics
↓
Understand System Performance
↓
Improve the Agent
This was another important shift in my thinking: building an AI agent is only one part of the problem; understanding how it performs is equally important.
9. Specialist Agent Handoff
One of my favorite features from the final stages of the challenge was introducing a specialist agent.
The idea is simple:
One agent does not need to be an expert at everything.
Instead of making the main agent responsible for every possible healthcare-related task, a specialized agent can take over when a request requires a narrower area of expertise.
The flow becomes:
┌── Main Agent ──→ Normal Request
│
User → Voice → Main Agent
│
└── Handoff ──→ Specialist Agent
↓
Continue Conversation
The important part is that the user should not have to repeat their entire problem.
The main agent can announce the transition, and the specialist can continue using the relevant conversation context.
This introduced me to an important concept in agentic AI:
Specialization can make an AI system more manageable, rather than trying to make one agent responsible for everything.
How the Complete System Fits Together
After combining the different capabilities, AarogyaMitra can be viewed as a collection of interconnected components.
┌──────────────────┐
│ User │
│ Voice Input │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Speech-to-Text │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Main Agent │
│ │
│ Personality │
│ Guardrails │
│ Memory │
│ Tool Routing │
└───────┬──────────┘
│
┌─────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌────────────┐ ┌──────────────┐
│ Memory │ │ Tools │ │ Human Help │
└─────────┘ └────────────┘ └──────────────┘
│
▼
┌──────────────────┐
│ Specialist Agent │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Response / LLM │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Murf Falcon │
│ Text-to-Speech │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ User hears reply │
└──────────────────┘
The real implementation contains more details, but this architecture captures the core idea.
The Technology Behind a Voice Agent
One of the biggest things I learned is that a voice AI agent is not just an LLM.
A real-time voice system typically combines several major components.
Speech-to-Text
Converts the user's spoken audio into text that the AI system can understand.
Large Language Model
Processes the user's request, maintains conversational reasoning, follows instructions, and decides whether additional actions are required.
Tools and Application Logic
Allow the agent to retrieve information, perform actions, or interact with external services.
Text-to-Speech
Converts the generated response back into natural audio.
For AarogyaMitra, this is where Murf Falcon plays an important role.
Real-Time Transport
A real-time communication layer connects the audio interaction and agent components so that the user can have an actual conversation instead of waiting for disconnected requests.
Some of the Challenges I Faced
The development journey was not completely smooth.
And honestly, that was one of the most valuable parts of the challenge.
Challenge 1: Connecting Multiple Components
A voice agent requires several systems to work together.
Speech recognition, the LLM, tools, TTS, real-time communication, and application logic all have to work together.
A problem in one component can affect the entire conversation.
What I learned
Instead of debugging everything at once, I learned to isolate the components and verify them individually before connecting them together.
Challenge 2: Environment Variables and API Configuration
Voice AI applications depend on external services and API credentials.
Managing these credentials correctly was important.
I learned to keep sensitive information inside environment variables rather than hard-coding keys into the project.
For example:
API_KEY=your_api_key_here
The actual key should remain private and should never be committed to GitHub.
Challenge 3: Making the Agent Know When to Escalate
An AI agent should not attempt to solve every problem.
Designing appropriate boundaries and escalation behavior required thinking beyond just generating a good response.
The solution was to make escalation part of the agent's decision-making process rather than treating it as an afterthought.
Challenge 4: Specialist Handoff
The final challenge was ensuring that the specialist agent could continue the conversation naturally.
A handoff is not useful if the user has to explain everything again.
The important lesson was to preserve the relevant conversation context and make the transition explicit to the user.
How You Can Start Building Your Own Voice Agent
If you want to build a voice agent yourself, start small.
You do not need to implement all of AarogyaMitra on day one.
A good progression is:
Step 1 → Basic voice conversation
Step 2 → Add an LLM
Step 3 → Add text-to-speech
Step 4 → Define the agent's personality
Step 5 → Add guardrails
Step 6 → Add tools
Step 7 → Add memory
Step 8 → Add human escalation
Step 9 → Add specialist agents
Step 10 → Add analytics and real-world workflows
Start with one simple use case and expand gradually.
Running the Project
The general setup process is:
1. Clone the repository
git clone YOUR_GITHUB_REPOSITORY_URL
cd YOUR_PROJECT_FOLDER
2. Install dependencies
Follow the dependency instructions included in the repository.
3. Configure environment variables
Create your .env file and add the required API credentials.
For example:
MURF_API_KEY=your_murf_api_key
LLM_API_KEY=your_llm_api_key
Use your own environment variable names based on the project configuration.
Never commit the **.env**** file or expose real API keys publicly.**
4. Start the application
Run the backend and frontend according to the instructions in the repository.
5. Connect to the voice agent
Open the application and start a conversation with AarogyaMitra.
Try two types of interactions:
Normal request
The main agent should handle it directly.
Specialized request
The main agent should recognize that another agent or workflow is more appropriate and perform the required handoff.
Evidence From My Build
Throughout the challenge, I captured different stages of the project.
Some of the evidence I would recommend including in this article:
[Add Screenshot: AarogyaMitra Main Interface]
[Add Screenshot: Voice Conversation]
[Add Screenshot: Tool / Agent Interaction]
[Add Screenshot: Human Escalation or Call Analytics]
[Add Screenshot or GIF: Specialist Agent Handoff]
[Add Final Demo Video]
These visuals make the architecture and user experience easier to understand than text alone.
What I Learned During These 10 Days
This challenge changed how I think about AI agents.
Before starting, it was easy to think about an AI assistant as:
User → Prompt → LLM → Answer
After building the project, I now see a voice agent more like:
User
↓
Audio
↓
Speech Recognition
↓
Agent
↓
Memory / Tools / Guardrails / Routing
↓
Specialists / Humans
↓
Response
↓
Text-to-Speech
↓
Audio
The LLM is only one part of the complete system.
I also learned that:
1. Voice UX matters
A technically correct answer is not enough. The interaction has to feel natural.
2. Guardrails are essential
Especially in sensitive domains, the agent needs clear boundaries.
3. Tools make agents more useful
An agent becomes significantly more capable when it can interact with real functionality instead of only generating text.
4. Memory changes the experience
Remembering relevant context can make conversations feel much more natural.
5. Human-in-the-loop systems are valuable
The goal should not always be full automation. Sometimes the best AI system is one that knows when a human should take over.
6. Multi-agent systems can improve specialization
Instead of making one agent responsible for everything, specialized agents can handle focused tasks.
7. Building is different from just learning
The biggest lesson was that concepts such as TTS, tools, memory, guardrails, handoffs, and analytics become much easier to understand when you actually have to make them work together.
What I Would Build Next
AarogyaMitra is still a starting point.
If I continue developing it, I would like to explore:
- More Indian language support
- Better handling of code-mixed conversations
- More healthcare-related tools
- Improved appointment workflows
- More specialized healthcare agents
- Better conversation analytics
- Improved accessibility
- Stronger safety and evaluation mechanisms
- More robust real-world testing
The long-term goal would be to make the system more reliable, accessible, and useful while maintaining appropriate safety boundaries.
Final Thoughts
The 10 Days of Voice Agents — VoiceForBharat Edition challenge was much more than learning how to make an AI speak.
It was a journey from building a basic conversational agent to understanding the broader architecture required for a useful voice AI system.
Over these 10 days, I worked with concepts including:
Voice → LLM → TTS → Tools → Memory → Guardrails → Human Escalation → Outbound Calls → Analytics → Specialist Agents
Each step introduced a new layer of complexity.
But the biggest takeaway for me is simple:
Building a good AI agent is not about making the AI do everything. It is about designing the right system around the AI.
AarogyaMitra is my attempt to apply that idea to healthcare access through voice.
I'm excited to continue exploring what voice AI can do for real-world problems, especially in India.
🔗 Project Links
GitHub Repository
https://github.com/Vamshikrishan/murf-livekit-ai-agent
Demo Video
https://drive.google.com/file/d/1_-pdLm-ugZ0IZ83ZxsZenJjcPbUNhC-K/view?usp=drive_link
Acknowledgements
A big thank you to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition and providing an opportunity to learn by building.
Special thanks for the resources and ecosystem around Murf Falcon, LiveKit, and the voice AI community.
This challenge gave me the opportunity to move beyond simply learning about AI agents and actually build one from the ground up.
10 Days. One Voice Agent. A Lot of Learning. 🚀
Built with curiosity, experimentation, and a lot of debugging.
#VoiceForBharat #MurfAI #VoiceAI #AIAgents #GenerativeAI #HealthcareAI #ArtificialIntelligence #MachineLearning #MurfFalcon #LiveKit
