From a Voice Bot to a Responsible Health Access Agent: Building Jana Seva in 10 Days
The Problem
Accessing healthcare information is not always as simple as opening a website and reading a page.
For many users, especially in contexts where typing, navigating websites, or understanding technical health terminology is difficult, voice can be a more natural interface.
That became the starting point for Jana Seva — a voice-first Health Access AI assistant built during the 10 Days of Voice Agents — VoiceForBharat Edition.
The goal was not to build an AI doctor.
The goal was to build a system that can help users navigate health-access information, use real data when appropriate, remember useful context, and know when it should stop and involve a human.
Meet Jana Seva
Jana Seva is designed as a real-time voice assistant for public-health access.
A user can talk naturally through the browser and ask about healthcare access, nearby health facilities, environmental health information, or appointment-related help.
The system can also:
- use live data tools;
- remember consented user context;
- make controlled outbound reminder calls;
- create human escalation requests;
- show call analytics;
- hand a conversation to a specialist agent.
The project uses Murf Falcon for the voice output and LiveKit Agents for real-time communication.
Why Voice?
Voice changes the interaction model.
Instead of:
Search → Read → Understand → Navigate → Decide
the experience can become:
Speak → Listen → Respond → Continue
That matters when a user is more comfortable speaking than typing or when a conversational interface is more natural for the task.
For an Indian Health Access use case, the ability to support English, Hindi and Hinglish interaction paths also makes the experience more conversational.
How the System Works
At a high level:
Browser / Phone
↓
LiveKit
↓
Voice Activity + Turn Detection
↓
Deepgram Nova-3
↓
Jana Seva Agent
↓
Google Gemini
↓
Tools / Memory / Escalation / Specialist
↓
Murf Falcon
↓
LiveKit
↓
User
The system separates real-time transport, speech recognition, reasoning, external tools, persistence and voice synthesis.
That separation made it possible to add new capabilities over the ten-day challenge without turning the entire agent into one large block of logic.
The Journey
Day 1 — Making the Agent Speak
The first milestone was the basic real-time voice pipeline.
The browser connects through LiveKit, speech is recognized, the agent reasons about the request, and Murf Falcon generates the response.
This established the foundation.
Day 2 — Defining What the Agent Should Do
The next step was defining the agent's role and boundaries.
The system was designed around Health Access rather than diagnosis.
That distinction became important later because it determined when the agent should answer, when it should use a tool, and when it should escalate.
Day 3 — Safety and Guardrails
Healthcare is not a domain where an AI should confidently improvise.
The agent therefore needed boundaries around diagnosis, prescription-like behavior and emergency situations.
The design principle became:
Be useful without pretending to be a doctor.
Day 4 — Memory
A returning user should not necessarily have to repeat the same basic context every time.
Jana Seva introduced consent-based caller memory so useful context could be reused by later workflows.
This became especially useful once the agent started using tools.
For example, a previously known district could help a health-facility lookup without forcing the user to repeat it.
Day 5 — Real-World Data
This was where the project started feeling less like a demo chatbot.
Jana Seva gained function-calling tools for:
Nearby health facilities
A facility lookup can search for:
- PHCs
- CHCs
- District Hospitals
- Jan Aushadhi stores
The system can use OpenStreetMap Nominatim and a local fallback registry.
Environmental health advisory
The agent can retrieve:
- AQI
- PM2.5
- PM10
- temperature
- respiratory precautions
using Open-Meteo.
The tools also have failure paths.
If an external service fails, the agent should not silently invent a result.
It should communicate the failure and use the available fallback where appropriate.
Day 6 — Outbound Calls
The next challenge was moving beyond inbound browser conversations.
Jana Seva added controlled outbound health-reminder workflows using Twilio and LiveKit SIP.
The use case was intentionally narrow:
- vaccination follow-ups;
- medication reminders.
The call begins with identification, purpose and an opt-out mechanism.
If the user says they do not want future calls, the agent immediately handles the opt-out path.
Automatic retries are also intentionally conservative.
This was an important lesson:
A voice system that can call people needs stronger boundaries than a system that only waits for users to call it.
Day 7 — Knowing When to Ask a Human
This became one of the most important parts of the project.
The agent should not try to solve every problem.
Two major escalation conditions were:
- red-flag symptoms / emergency situations;
- requests for diagnosis or medical decisions outside the agent's safe scope.
The workflow is:
Detect
↓
Explain
↓
Ask permission
↓
Summarize
↓
Create request
↓
Generate reference ID
↓
Tell user what happens next
The human receives a concise operational summary rather than an unnecessary full transcript.
The escalation dashboard organizes requests by urgency and status.
This changed my understanding of "smart" agents.
A smart agent is not necessarily the one that answers everything.
Sometimes the smarter action is:
"This is outside what I should handle. Let me get the right human involved."
Day 8 — Measuring the Agent
Once an agent is doing real work, I need to know whether it is actually succeeding.
Jana Seva therefore gained a call analytics dashboard backed by SQLite.
The required metrics are:
- Total Calls
- Successful Calls
- Failed Calls
The system also calculates success rate and shows operational call history.
A successful call means the intended health-access task was safely completed, or a required human escalation was correctly completed with user permission.
A failed call means the intended outcome was not reached.
This distinction matters because a "failed" call does not necessarily mean the software crashed.
A user hanging up before completing the task can also be a failed outcome.
Day 9 — Handing the Conversation to a Specialist
The final major agent capability was specialist routing.
Instead of building one huge agent that tries to know everything, Jana Seva now has a focused:
Clinic & Appointment Specialist
The main agent detects requests related to:
- clinic discovery;
- department navigation;
- appointment assistance.
It announces the handoff first.
Then it transfers useful context so the user does not need to repeat the entire request.
For example:
User:
"I want to find a clinic and get help with an appointment."
↓
Main Jana Seva Agent:
"I'll connect you with our Clinic & Appointment Specialist."
↓
Clinic & Appointment Specialist:
"I understand you're looking for help with a general health consultation appointment..."
The specialist has its own scope and safety boundaries.
Emergency red flags bypass the specialist and use the human escalation workflow instead.
The Technical Stack
| Layer | Technology |
|---|---|
| Real-time transport | LiveKit |
| Voice synthesis | Murf Falcon |
| Speech recognition | Deepgram Nova-3 |
| LLM | Google Gemini |
| Backend | Python |
| Frontend | Next.js / React |
| Database | SQLite |
| Telephony | Twilio + LiveKit SIP |
| Facility lookup | OpenStreetMap Nominatim |
| Environmental data | Open-Meteo |
The Hard Parts
1. Designing safe boundaries
The easiest mistake is to make a healthcare agent sound too confident.
The system needed explicit boundaries around diagnosis and emergency situations.
The solution was to make escalation a first-class workflow rather than an afterthought.
2. Handling external data failures
Live APIs can fail.
A voice assistant cannot simply show an error stack trace to a user.
The tool layer therefore needed fallback behavior and a spoken failure path.
The agent should communicate that the live lookup was unavailable rather than pretending that stale or invented information is current.
3. Making handoffs feel natural
A bad multi-agent handoff would sound like:
"Please repeat everything to the next agent."
That defeats the purpose.
The specialist handoff therefore transfers the relevant intent and context so the conversation can continue naturally.
4. Knowing what to measure
A dashboard is only useful if the metrics represent the actual objective.
Instead of measuring only technical uptime, Jana Seva records whether the call reached its intended outcome.
That makes the analytics more useful for evaluating the actual agent experience.
How Someone Else Can Build a Similar Voice Agent
The architecture can be reduced to four core components:
1. Speech-to-text
Convert the user's speech into text.
2. LLM
Reason about the request and decide what should happen.
3. Text-to-speech
Turn the response back into natural speech.
4. Real-time transport
Move audio between the user and the agent with low latency.
LiveKit handles the real-time layer in Jana Seva.
Murf Falcon handles speech synthesis.
Deepgram handles speech recognition.
Google Gemini handles reasoning and tool calling.
Running Jana Seva
Prerequisites
Install:
- Python 3.10+
- uv
- Node.js 18+
- pnpm
- LiveKit
Then configure the required environment variables locally.
Example:
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key
Never commit real credentials.
Install backend dependencies:
cd backend
uv sync
uv run python src/agent.py download-files
Install frontend dependencies:
cd frontend
pnpm install
Start the project using the included startup script or run the LiveKit server, backend agent and frontend in separate terminals.
Then open:
http://localhost:3000
Allow microphone access and start a conversation.
Evidence From the Build
The project includes screenshots showing:
- the Jana Seva homepage;
- an active voice session;
- the human escalation command center;
- the call analytics dashboard;
- a specialist-agent handoff.
The project also includes demonstration videos from the challenge stages.
What I Would Build Next
If I continued developing Jana Seva, I would focus on:
- verified government health-service integrations;
- stronger multilingual and code-mixed conversations;
- more specialist agents;
- improved escalation workflows;
- duplicate escalation detection;
- resolution callbacks;
- production authentication;
- stronger safety evaluations;
- better observability and latency measurement.
Final Takeaway
The biggest lesson from these ten days was that building a voice agent is not just about making an AI that can talk.
It is about designing a system that knows what it can do.
It needs to know:
- when to answer;
- when to use a tool;
- when to remember;
- when to ask permission;
- when to stop;
- when to involve a human;
- and when another specialist should take over.
That is the idea behind Jana Seva.
Healthcare, in your voice.
Links
Source code: https://github.com/lightcode01-oss/murf-livekit-starter/tree/day10
Built for
10 Days of Voice Agents — VoiceForBharat Edition
Powered by Murf Falcon and LiveKit Agents.
