Building Saathi: A Voice Agent for Health Access in Rural India

Millions of people in rural and semi-urban India don't have easy access to basic health guidance. A simple question — "is this fever serious?", "where's the nearest hospital?", "do I need to see a doctor?" — often means a long trip, a language barrier, or simply not knowing where to start. Most digital health tools assume typing, reading English, and a smartphone app. That leaves out exactly the people who need help most.

Voice fixes this. No typing, no app literacy required, and — critically — it can speak the way people actually talk: Hindi, English, or a natural mix of both.

That's what I built over 10 days as part of 10 Days of Voice Agents — VoiceForBharat Edition: Saathi, a voice assistant for the Health Access track.

What Saathi does

Saathi is a voice agent a caller can talk to about everyday health concerns. It:

Gives safe, general guidance on whether a symptom needs home care, a doctor visit, or urgent attention
Never diagnoses and never names a medicine — it stays firmly inside what's safe for a non-medical assistant to say
Remembers returning callers (with consent) and continues from the last conversation
Looks up real facility information (hours, walk-in policy) through a dedicated specialist agent
Recognizes exactly when a situation is beyond it, and hands off to a human health worker — with the caller's permission
Tracks its own performance on a live dashboard

How the system works

At a high level, every voice agent — including Saathi — is four components wired together in a loop:

Speech-to-text (STT) — turns the caller's voice into text (Saathi uses Deepgram's nova-3 model, set to language="multi" so it correctly picks up Hindi as well as English)
LLM — the "brain" that decides what to say and which tools to call (Saathi uses Gemini)
Text-to-speech (TTS) — turns the reply back into voice (Murf Falcon — genuinely the fastest TTS I've used building this; the low latency is what makes the conversation feel like a real back-and-forth instead of a request-response bot)
Real-time transport — the pipe that carries audio both ways with low enough latency for a natural conversation (LiveKit)

On top of that pipeline, Saathi has:

A SQLite database for caller memory, escalation requests, and call outcomes
Function tools the LLM calls itself — a rule-based triage classifier, a facility lookup, an escalation creator — instead of guessing answers
A second, specialist agent (Clinic & Appointment Specialist) that the main agent hands the conversation to for logistics questions, without the caller repeating themselves
A Discord webhook integration so human escalations land somewhere real, with a reference ID
A small Flask dashboard showing total/successful/failed calls, built from real call data

The most important features

An Indian voice, and a language-aware pipeline. Saathi speaks in an Indian English/Hindi voice via Murf Falcon, and the prompt explicitly requires Hindi to render in Devanagari script (नमस्ते), never romanized — a detail that's easy to miss and immediately makes the agent feel less robotic.

A personality with hard limits, not just a system prompt. Saathi's instructions are structured as IDENTITY → OBJECTIVES → KNOWLEDGE → LANGUAGE → GUARDRAILS → STYLE. The guardrails aren't suggestions — they're enforced behaviorally: no diagnosis, no drug names, and a defined escalation script for red-flag symptoms.

Memory with consent as a hard rule, not a nice-to-have. Saathi asks permission before saving anything about a caller, and if they say no, nothing is stored. For a health use case, this isn't optional.

Tools that compute real answers instead of the LLM guessing. The triage classifier is a deterministic, rule-based function — not the LLM freestyling a medical opinion. It also has a defined failure path: if a tool errors out, Saathi says so out loud and falls back to safe generic advice, instead of going silent or making something up.

Escalation with a real destination. When Saathi hits its limits, it creates a tracked request — sent to a real Discord channel via webhook — with a reference ID the caller can hold onto, after explicit consent.

A specialist handoff. Saathi stays a generalist for health guidance; a separate Clinic & Appointment Specialist agent takes over for facility-logistics questions, introduces itself, and hands the conversation back if the topic shifts back to a health question.

The hardest part: it wasn't the AI, it was the plumbing

If I'm honest, the single biggest time sink across all 10 days wasn't prompt design or tool logic — it was local network/DNS reliability. The LiveKit worker would register successfully, then intermittently fail to establish the actual room connection with a getaddrinfo failed error, or the local Python process would go unresponsive under system load and get killed and restarted by the worker supervisor.

What I learned from debugging this over and over:

A startup that takes 40-60+ seconds instead of ~7-10 seconds is an early warning sign of system load about to cause a connection drop — that gap became my signal to close other apps and retry
Switching DNS to 8.8.8.8 / 8.8.4.4 helped more than anything else
When a call genuinely can't connect in time, the honest move is to show your working code and explain the network issue plainly, rather than pretend it worked

The second real difficulty was environment variable duplication between the frontend and backend — LiveKit credentials need to match exactly in both .env.local files, and a single mismatched key produces a generic "invalid API key" error that doesn't tell you which key is wrong. Double-checking both files side by side, line by line, was the fix every time.

How to build your own

1. The four core pieces you need:

An STT provider (Deepgram, etc.)
An LLM (Gemini, OpenAI, etc.)
A TTS provider — I'd recommend Murf Falcon for latency
A real-time transport layer (LiveKit is the easiest path)

2. Setting up and running the project:

Fork the Murf LiveKit Starter repository
Follow its README for uv sync (backend) and pnpm install (frontend)
Run the backend with uv run python src/agent.py dev and the frontend with pnpm dev

3. Where API keys go — and where they must NOT go:

Copy .env.example to .env.local in both backend/ and frontend/ — never commit .env.local, it's already gitignored
You'll need: a LiveKit Cloud project (URL, API key, secret), a Murf API key, a Deepgram API key, and an LLM provider key
Never paste real keys into chat, screenshots, or your repo — treat them like passwords

4. Connecting and testing:

With both the backend and frontend running, open localhost:3000, click the start button, allow microphone access, and talk
Watch the backend terminal for a "registered worker" log line as confirmation the agent is live

5. The code:
My repository: https://github.com/rohitkumar31/murf-livekit-starter

What I'd improve next

Swap the local rule-based triage classifier for a reviewed, larger medical red-flag dataset
Replace the hand-built facility dataset with a real state health department API
Add outbound calling fully (Twilio + LiveKit SIP) for proactive follow-up reminders
Add automatic language detection confidence scoring to catch mis-transcriptions in noisy environments