I built a Multilingual healthcare voice agent that knows when call or shutup.

Ten days ago I set out to build a voice agent for the Murf 10 Days of Voice Agents — VoiceForBharat challenge. I picked the Health Access track and built Careva: a phone-and-browser helpline that helps someone in India find the nearest working health facility, understand a government scheme, ask what a medicine actually costs, and — when it matters — stop talking and get a human involved.

This post is the story plus the guide. Everything here is in the repo: https://github.com/ace-ify/murf-livekit

1. The problem and the users

The person I built for does not have a problem that a website solves.

They have a fever at 9pm and don't know if the PHC is open. They have a ₹40 prescription and no idea the generic salt costs ₹6 at a Jan Aushadhi store. They're eligible for Ayushman Bharat and have never read the eligibility page, because the eligibility page is in English, is 2000 words long, and assumes a smartphone with data.

They can talk, though. Voice is not a nicer interface here — it's the only interface that clears the bar. No app install, no typing in a script your keyboard doesn't have, no literacy assumption. A phone call works on a ₹1500 feature phone in a village with 2G.

So the design constraint was: it has to work when spoken, in Hindi, on a bad line, to someone who is scared.

2. What the agent does

Careva answers in the language you spoke to her in. She can:

• find the nearest PHC / CHC / district hospital with OPD timings, from a district name or a pincode
• explain PM-JAY, JSSK and immunisation schedules from a local knowledge base
• look up the generic salt for a branded medicine and what you'd save
• give an air-quality advisory for a district (useful for asthma and elderly callers)
• remember you between calls, if you say it's okay
• call you — medication and vaccination reminders
• escalate to a human health worker with a spoken reference number
• hand you to Samar, a specialist agent, for detailed clinic/appointment questions
• and above all: recognise a medical emergency and say "call 108 now" as the first sentence, before any greeting or name

That last one drove most of the engineering.

3. How the system works

🎙️ caller ──audio──▶ Deepgram nova-3 (multi) ──text──▶ LLM ──text──▶ Murf Falcon ──audio──▶ 🔊 caller
                                    ▲                                                │
                                    └────────────── LiveKit (WebRTC / SIP) ──────────┘

Four moving parts, and you can swap any of them:

The session, more or less verbatim from backend/src/agent.py:

session = AgentSession(
    stt=deepgram.STT(model="nova-3", language="multi"),
    llm=FallbackAdapter([...], attempt_timeout=15.0),
    tts=murf.TTS(
        voice=MURF_VOICE,                 # "Anisha"
        style="Conversational",
        tokenizer=CleanSentenceTokenizer(min_sentence_len=4),
        text_pacing=True,
    ),
    turn_detection=MultilingualModel(),
    vad=ctx.proc.userdata["vad"],
    preemptive_generation=True,
    user_away_timeout=SILENCE_TIMEOUT,    # 12.0
)

Two small lines with big effects. preemptive_generation=True starts the LLM on a partial transcript, which removes a visible chunk of the reply gap. And noise cancellation is picked per participant — BVCTelephony() for SIP callers, BVC() for browser — because phone audio is band-limited and the generic model does worse on it:

noise_cancellation=lambda params: (
    noise_cancellation.BVCTelephony()
    if params.participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP
    else noise_cancellation.BVC()
),

4. The features that mattered

Tools that hit real APIs, with a floor under them. Facility lookup is OpenStreetMap Nominatim + the India Post pincode API. Medicines are NLM RxNorm. Air quality is Open-Meteo. Schemes are a hand-rolled BM25 index over local markdown — no vector DB, ~150 lines, and it retrieves fine over a corpus this size. Every external call has a 2.5–3.5s budget and degrades to a curated fallback registry of verified facilities. The one thing the agent is never allowed to say is "I can't help" — the last fallback is still 108 and the nearest hospital.

Consent-gated memory. A callers table in SQLite keyed on the participant identity (the phone number, for SIP). It stores name, district, age band, ongoing conditions, last triage outcome — and only after the caller says yes. forget_caller hard-deletes. On reconnect the greeting is rebuilt from that record: "नमस्ते प्रिया जी… पिछली बार हमने बुखार के बारे में बात की थी। आज आप कैसा महसूस कर रहे हैं?"

Escalation that a human can actually work. The agent asks permission first — "May I send a short summary to a human health worker?", in the caller's own language — and a "no" writes literally nothing. What gets stored is six fields, never the transcript, and it goes through one PII scrub choke point so storage and the webhook can't diverge. The caller hears a reference number, ESC-0007, read digit by digit.

Specialist handoff. transfer_to_clinic_specialist swaps the agent object on the live session and swaps the Murf voice with it, so the caller hears a different person (Anisha → Samar, female → male) rather than the same voice claiming to be someone else. State carries across: caller facts, call id, turn counters, and the emergency flags.

A dashboard that defines its own metric. /admin shows total / successful / failed / no-answer calls, a failure breakdown, and a success rate that excludes in-progress calls from the denominator. The header states the definition out loud: success = caller received triage guidance or was escalated to a human. A success rate whose definition lives only in someone's head is a vanity number.

5. The hard parts (all four of these were real bugs)

The agent said goodbye to a heart attack

Worst thing I found in testing. Caller: "my friend is having chest pain". Careva gave the 108 line correctly, and then a few turns later cheerfully wrapped up — "aapka din shubh ho" — and hung up.

The prompt already said don't do that. Prompts are suggestions.

The fix was to move safety out of the model entirely. A regex scans every final transcript — in English, Roman Hinglish, and Devanagari — and latches a flag. end_call then refuses to run:

# Hard guard: the LLM once said goodbye to a caller whose friend was having a
# heart attack. A prompt line is not enough — refuse the hangup outright.
if self.emergency_flag and not self.escalation_created_flag:
    logger.warning("end_call BLOCKED: unhandled emergency (reason=%r)", reason)
    return "REFUSED — you cannot end this call. ..."

The same guard blocks the specialist handoff. Lesson: if a behaviour is safety-critical, it belongs in code that the model calls, not in text the model reads. The prompt gets you the good path; the guard gets you the bad one.

The bad accent wasn't the voice, it was the locale

English replies sounded wrong — an Indian-English sentence being pushed through a Hindi voice, mangling the vowels. I spent a while rewriting the prompt, which was the wrong layer entirely.

Deepgram's multi model tags every final transcript with the language actually spoken. So: read that tag, and update the TTS locale mid-call.

session.tts.update_options(locale=locale)   # "hi" or "en", short-circuited if unchanged

Locale fixes the accent. The prompt fixes word choice. Two different bugs that sound like one.

"consent_given": "true" killed entire turns

Consent params in my tools are typed str, not bool, and that looks like a mistake until you've seen this: Groq's Llama 3.3 intermittently emits the string "true" for a boolean parameter, Groq rejects it server-side with tool_use_failed, and the whole turn dies — mid-sentence, on a health call.

A string parameter cannot be malformed. So it's a string, parsed fail-closed, and the truthy set speaks Hindi:

return str(val).strip().lower() in ("true", "yes", "haan", "haan ji", "1", "y")

Anything unrecognised means no consent. Failing closed on a consent check is free; failing open is a privacy incident.

Every call looked like a silent disconnect

My analytics said 100% no-answer. The calls were fine. I was subscribed to user_speech_committed and agent_speech_committed, which don't exist in livekit-agents 1.4 — and a handler for an event that never fires is completely silent. conversation_item_added is the real one.

Two more of the same species, both fixed: finalising the call row on the session close event lost a race with the loop shutting down and left rows stuck at in_progress (use ctx.add_shutdown_callback, which LiveKit awaits); and dispatch metadata was being parsed after record_call_start(), so that call raised a swallowed NameError and recorded zero calls. Wire up your telemetry and then verify it with your own eyes, because broken instrumentation reports success.

6. Build your own — the short version

Start from Murf's LiveKit starter. You need Python 3.10+, uv, Node 18+, pnpm, and a free LiveKit Cloud project.

git clone https://github.com/ace-ify/murf-livekit.git && cd murf-livekit
cd backend && uv sync && uv run python src/agent.py download-files
cd ../frontend && pnpm install

Keys. Copy .env.example → .env.local in both backend/ and frontend/. .env* is gitignored — keep it that way, and never paste a key into a chat window, an issue, or a screenshot. You need LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET, MURF_API_KEY (murf.ai/api/dashboard), DEEPGRAM_API_KEY, and GOOGLE_API_KEY. In production these are platform env vars (Railway for the agent, Vercel for the UI) — same LiveKit project on both.

Run and talk to it.

uv run python src/agent.py console          # talk in your terminal, no frontend needed
uv run python src/agent.py dev              # + pnpm dev, then open localhost:3000

console mode is the fastest loop you'll get — it's where I did most of the debugging.

Then write one tool. Not five. A tool is just a decorated async function; the docstring is the spec the model reads, so spend your time there, not on the code:

@function_tool
async def find_nearest_health_facility(
    self, context: RunContext,
    location_or_pincode: str = "", facility_type: str = "any",
) -> str:
    """Find the nearest Primary Health Centre (PHC), Community Health Centre (CHC), or Hospital.

    Use this tool when a caller asks:
    - "Mera paas ka PHC/hospital kahan hai?" or "Where is the nearest health centre?"
    - "OPD kitne baje tak khula hai?" or "What are the hospital timings?"
    ...
    Args:
        location_or_pincode: District name, city name, area, or 6-digit Indian PIN code.
            If empty, automatically checks caller memory.
    """

Note the example questions in both languages — that's not documentation, that's routing. And note the last line: leaving the location empty and resolving it from memory is what makes a second call feel like a continuation instead of a form.

7. What I'd do next

• Cut turn latency properly. I log reply = llm_ttft + tts_ttfb per turn but I haven't done a real p50/p95 pass across the fallback chain. (One thing I did measure: NIM's llama-3.3-70b-instruct is listed but never responds — read timeout past 45s — while 3.1-70b gives ~0.8s TTFT with working tool calls. Measure your fallbacks; a dead fallback is worse than none, because you'll trust it.)

• Put real auth on /admin. It lists health complaints. Status writes need ADMIN_TOKEN and fail closed without it, but the GET is open for local dev. That page does not go public as-is.
• Escalation triage instead of keyword matching. The current gate is a word list that errs open. It works; it isn't triage.
• Widen the fallback facility registry — the live-API-fails path is the path a rural caller is most likely to hit.

8. Links

• Code: https://github.com/ace-ify/murf-livekit
• Murf Falcon docs · Murf voice library
• LiveKit Agents · Deepgram Nova-3

If you build one thing from this post, build the guard, not the prompt. The model will be charming and wrong at some point, and on a health line "charming and wrong" has a cost. Put the floor in code.

#VoiceForBharat