Building HealthSaathi: My 10-Day Journey Into Voice Agents

Introduction

Over the past 10 days, I took part in the 10 Days of Voice Agents — #VoiceForBharat Edition, where I built a voice agent from the ground up.

For my project, I chose the Health Access track and built HealthSaathi, a voice-based health assistant designed to make basic health guidance more accessible through natural conversations.

The goal wasn't just to make an AI that could answer questions. Throughout the challenge, I worked on making HealthSaathi more useful in a real-world setting by adding memory, tools, safety guardrails, outbound calling, and human escalation.

This journey also taught me that building a voice agent involves much more than connecting an LLM to a microphone.

The Problem I Wanted to Solve

Healthcare can sometimes feel difficult to navigate, especially when someone doesn't know where to begin or simply wants basic guidance.

I wanted to explore how voice could make this interaction simpler.

Instead of requiring users to type everything into a chatbot, HealthSaathi allows them to communicate through conversation. The idea is to make the experience feel more natural and accessible, especially for users who may prefer speaking over typing.

HealthSaathi is not designed to replace doctors or provide professional medical diagnosis. Its purpose is to assist users with basic health-related guidance and direct them toward appropriate next steps when necessary.

What is HealthSaathi?

HealthSaathi is a voice-based AI assistant that I built for the Health Access track of the challenge.

The agent can:

• Have real-time voice conversations
• Respond using Murf Falcon TTS
• Remember relevant information about returning users
• Use tools to perform useful tasks
• Make outbound follow-up calls
• Follow safety and conversation guardrails
• Escalate conversations when AI assistance is not enough
• Track the state of the conversation through the frontend

The project gradually evolved from a simple voice conversation into a more complete voice-agent system.

How the System Works

At a basic level, the system follows this flow:

User speaks → Speech-to-Text → LLM → Memory/Tools → Response → Murf Falcon TTS → User hears the response

The different components have different responsibilities.

Speech-to-Text (STT) converts the user's speech into text so that the language model can understand it.

LLM processes the user's request, maintains the conversation, decides how to respond, and determines when tools are needed.

Memory allows HealthSaathi to retain relevant information instead of treating every conversation as completely new.

Tools allow the agent to perform specific tasks instead of relying only on the LLM's generated response.

Murf Falcon TTS converts the response back into speech so that the user can hear the answer naturally.

LiveKit handles the real-time communication layer between the user and the voice agent.

The Important Features I Built

1. Voice with Murf Falcon

Voice is the core of HealthSaathi.

For the text-to-speech layer, I used Murf Falcon, which the challenge describes as its fastest TTS API.

Using a dedicated TTS system made the interaction feel much more natural than simply displaying text responses.

This was also an important part of building a voice-first experience for the #VoiceForBharat challenge.

2. Conversation States

I also personalized the frontend to show what the agent was currently doing.

The interface has states such as:

• Ready
• Connecting
• Listening
• Speaking
• Call ended

This made it easier for the user to understand whether the agent was listening, processing, speaking, or disconnected.

I also added microphone permission handling so that users receive feedback when the browser cannot access the microphone.

4. Tools

Another important step was giving HealthSaathi access to tools.

Instead of asking the LLM to generate every answer itself, tools allow the agent to perform specific actions or retrieve useful information.

This makes the system more reliable because certain tasks can be handled by deterministic functions rather than relying entirely on generated text.

The tools were integrated into the agent while keeping the existing voice pipeline intact.

5. Outbound Calling

One of the biggest steps in the challenge was moving beyond browser-based conversations.

Instead of always waiting for a user to open the application and start a conversation, I built an outbound calling flow for health follow-ups.

I used LiveKit telephony and Linphone to test the phone-side interaction.

The call begins by telling the person who is calling and why they are being contacted, and then gives them the option to continue the conversation.

This made the project feel much closer to a real-world voice application.

The Challenges I Faced

Building the project wasn't always straightforward.

One of the biggest problems I faced happened while working on the outbound agent.

During one call, the LLM tried to call an end_call tool even though that tool wasn't included in the tools available for that particular request.

The result was a tool-call validation error.

Instead of changing the whole architecture, I investigated how tools were being passed to the LLM and focused specifically on the outbound agent's tool behaviour.

This taught me an important lesson:

An LLM can only reliably call tools that are actually available to it in the current request.

It also showed me that voice agents require careful handling of tool availability and conversation logic. A normal chatbot might simply display an error, but in a live phone conversation, these failures directly affect the user experience.

Another Lesson: A Voice Agent Is More Than an LLM

Before starting this challenge, it was easy to think of a voice agent as:

Speech → LLM → Speech

But after building HealthSaathi, I realized that there are many more pieces involved.

A useful voice agent needs:

• Low-latency communication
• Speech recognition
• Natural text-to-speech
• Conversation logic
• Memory
• Tools
• Safety guardrails
• Error handling
• Telephony when phone calls are involved
• Human escalation when necessary

All of these pieces have to work together.

How Someone Can Build Their Own Voice Agent

If you want to build a similar project, the basic architecture can be broken into four main parts.

1. Speech-to-Text

You need a system that converts the user's speech into text.

2. Large Language Model

The LLM understands the user's request and generates the response or decides when a tool should be used.

3. Text-to-Speech

The generated response is converted back into audio. In my project, I used Murf Falcon for this part.

4. Real-Time Transport

You need something that can handle the real-time audio communication between the user and the agent. My project uses LiveKit for this layer.

Once these pieces are connected, you can start adding things like memory, tools, guardrails, and telephony.

Running HealthSaathi

The source code for the project is available on GitHub.

GitHub:
👉 https://github.com/sakshichhetri845-sys/10_days_murfaibuildathonn

What I Would Improve Next

Although HealthSaathi has come a long way during these 10 days, there is still a lot I would like to improve.

Some of the things I would explore next are:

• Better support for Indian languages and code-mixed conversations
• More robust health-related tools
• Better conversation evaluation
• Improved response latency
• More detailed call analytics
• More specialist-agent handoffs
• Better handling of complex conversations
• More extensive real-world testing

The challenge gave me a working foundation, but there is still plenty of room to make the system more reliable and useful.

What I Learned

The biggest thing I learned during these 10 days is that building a voice agent isn't just about making an AI talk.

It is about designing an entire interaction.

I learned how memory changes the way an agent can interact with returning users, how tools extend what an LLM can actually do, how telephony introduces a completely different set of challenges, and how small tool-calling or latency issues can have a big impact on a real conversation.

Most importantly, I learned by actually building, breaking, debugging, and rebuilding.

HealthSaathi started as an idea for a voice-based health assistant, but over these 10 days it became a much more complete system.

Project Links

GitHub:
👉 https://github.com/sakshichhetri845-sys/10_days_murfaibuildathonn

LinkedIn:
👉 https://www.linkedin.com/in/sakshi-chhetri-a962b6367/

Challenge:

10 Days of Voice Agents — #VoiceForBharat Edition

The 10 Days of Voice Agents — #VoiceForBharat Edition was a great experience because it pushed me to think beyond simply building an AI application.

I had to think about the user experience, memory, tools, safety, real-time communication, phone calls, and what happens when things don't work as expected.

I'm happy with what HealthSaathi became in these 10 days, and I'm excited to keep improving it beyond the challenge.

Thanks for following along with my HealthSaathi journey. 🎙️