I built a voice agent that explains money to people who were never taught it

I built a voice agent that explains money to people who were never taught it — here's how

My 10 Days of Voice Agents — VoiceForBharat Edition, and how you can build one too.

The problem, and who it's for

A huge number of people in India are using a bank account, UPI, and government schemes for the first time in their lives. The information they need exists — but it lives in English PDFs, dense scheme portals, and bank jargon. The people who most need it are often the least comfortable reading it, and they're the exact people fraudsters target with fake "OTP" and "KYC" calls.

Text-first apps quietly exclude these users. A voice helpline doesn't. You call, you talk in the words you actually use — including mixing a little Hindi into your English — and someone patient explains one idea at a time. No app to install, no form to read, no typing.

So for the Financial Services track, I built Dhan Saathi ("wealth companion") — a warm, voice-first money guide for everyday people in India. It explains banking basics, saving, UPI safety, and government schemes in plain words, and it is very careful about the one thing that matters most in finance: it never asks for your OTP or PIN, and it never pretends to be your bank.

What Dhan Saathi actually does

• Explains how a savings account, UPI, or a scheme works — one short idea at a time, out loud.
• Warns about common frauds and gives the real escalation path (call your bank, cyber helpline 1930).
• Remembers you between calls (with your permission) so you don't repeat yourself.
• Hands you to a scheme specialist when you ask about government schemes.
• Can call you back for a reminder, and can raise a request for a real human when something is beyond a helpline.
• Logs every call to an analytics dashboard so I can see what's working.

All of it in an Indian English voice, powered by Murf Falcon 2 — which mattered more than I expected (more on latency below).

How the system works

A voice agent is really four moving parts wired together in real time:

🎙️ You speak
   │  audio
   ▼
Deepgram STT (nova-3, "multi")   ← ears: speech → text, Hindi + English
   │  text
   ▼
Gemini (LLM)                     ← brain: decides what to say / which tool to call
   │  response text
   ▼
Murf Falcon 2 TTS (en-IN-anisha) ← voice: text → natural Indian-English speech
   │  audio
   ▼
LiveKit (real-time transport)    ← the pipe carrying audio both ways
   │
   ▼
🔊 You hear the reply

• STT (ears): Deepgram Nova-3 in "multi" mode, so it transcribes code-mixed Hindi/English, not just English.
• LLM (brain): Google Gemini. This is where the personality, guardrails, and tool-calling live.
• TTS (voice): Murf Falcon 2, voice en-IN-anisha, "Conversation" style.
• Transport: LiveKit Agents carries audio in both directions and handles turn detection, so the agent knows when you've stopped talking. The same transport works for a browser mic and a real phone call over a SIP trunk.

The whole backend is a single long-lived Python process. The frontend is a Next.js app. Neither calls the other directly — they both connect to the same LiveKit room, and LiveKit is the meeting point.

The features that tell the story

1. A personality with hard safety rails

The single most important file is the system prompt. I structured it as IDENTITY / OBJECTIVES / KNOWLEDGE / LANGUAGE / GUARDRAILS / STYLE so the agent has a clear job and clear limits. The guardrails are non-negotiable and read like a finance-helpline code of conduct:

• Never ask for or accept an OTP, PIN, CVV, or account number — and stop the caller if they start to share one.
• Never promise a loan or scheme will be approved.
• Never quote an interest rate or scheme figure from memory.
• Never move money or claim to access an account.

The agent even opens the call by promising it will never ask for your OTP — turning a guardrail into a trust signal.

2. Indian voice + code-mixed language, written in its own script

Beyond the Indian-English Falcon voice, the prompt lets the agent mix a little Hindi back when the caller does, and switch fully if they do. One rule that took real work: every language must be written in its own native script — Hindi in Devanagari (नमस्ते), never romanized "namaste" — because Falcon pronounces native script correctly but mangles romanized Hindi.

3. Memory for returning callers

A tiny SQLite store remembers a caller by a slug of their name. On the next call, the agent greets them and picks up where they left off ("last time you asked about Jan Dhan — how did that go?"). Two things make it safe and useful:

• Permission first. It only saves after explicitly asking, and only saves plain facts (a scheme they asked about, an age band) — never a sentence, never a number.
• A hard backstop. Even if the model slips, a sanitizer drops any value containing a long digit sequence before it can ever be written:

_LONG_DIGITS = re.compile(r"\d{6,}")  # looks like an account / card / OTP number

def _sanitize_facts(facts: dict) -> dict:
    clean = {}
    for key, value in (facts or {}).items():
        if _LONG_DIGITS.search(str(value)):
            logger.warning("Refusing to store sensitive-looking fact: %r", key)
            continue
        clean[str(key)] = str(value)
    return clean

4. A specialist handoff

The main guide deliberately does not answer government-scheme detail. The moment you ask about a scheme, it hands you to Yojana Mitra, a second agent with a narrower, sharper job and its own check_scheme_eligibility tool. In LiveKit Agents, a handoff is beautifully simple — a tool just returns another Agent, and the whole conversation carries over so you never repeat yourself:

@function_tool
async def transfer_to_scheme_specialist(self, context: RunContext):
    """Hand the caller to Yojana Mitra, the scheme specialist."""
    return SchemeSpecialist(
        chat_ctx=self.chat_ctx,   # the entire conversation carries over
        call_id=self._call_id,    # so success still attributes to this call
    )

When the scheme question is done, the specialist hands the call back — and the main guide picks up mid-conversation instead of re-greeting from scratch.

5. Outbound calls, human escalation, and a dashboard

• Outbound: a dispatcher can make Dhan Saathi call you — over a real SIP trunk — to deliver a reminder, with opt-out handling ("don't call again") baked in.
• Human escalation: for fraud or a dispute that a helpline can't settle, the agent asks permission and raises a structured request for a real person, then reads back a reference id.
• Analytics: every call is recorded with an outcome — success (an eligibility check or a human handoff happened), failed, or no_answer/busy/declined for outbound — and served on a small live dashboard so I can see success rate by channel.

The parts that were genuinely hard

Falcon was fast — my own config was what added lag

My first instinct was to "help" the TTS by setting a minimum sentence length and text pacing so speech would sound more deliberate. It did the opposite: it chopped the audio into fragments and added latency. The fix was to delete my cleverness and trust Falcon's default streaming tokenizer:

tts = murf.TTS(voice="en-IN-anisha", style="Conversation")
# no min_sentence_len, no text_pacing — those fragment the audio and add lag

Lesson: with a TTS this fast, the pipeline is not your bottleneck — your own "optimizations" often are. Measure before you tune.

The model kept romanizing Hindi

Early on, when a caller spoke Hindi, the LLM would reply with "namaste" instead of "नमस्ते" — and Falcon pronounced the romanized version poorly. Coaxing it in the general instructions wasn't enough. I had to add an explicit, separate LANGUAGE & SCRIPT rule stating every language must be written in its native script, with an example and an anti-example. That specificity is what made it stick.

Making memory trustworthy, not just functional

Getting the agent to save something was easy. Getting it to save the right thing was not. It would try to store whole sentences ("caller asked to save Jan Dhan") as a fact value. I fixed this in two layers: a tightened prompt that says "pass only the real fact, never a sentence or an instruction," and the digit-sanitizer backstop above. In finance, "the prompt usually behaves" is not good enough — you need a mechanical guarantee for the sensitive cases.

Build your own — a practical starting point

You genuinely can stand up a talking agent in an afternoon. Here's the shape.

The four components you need: STT (speech→text), an LLM (the brain), TTS (text→speech), and real-time transport (LiveKit) to move audio both ways with turn detection. Start from the Murf LiveKit Starter.

1. Get the code and install:

git clone https://github.com/praveenraj027/murf-livekit-starter
cd murf-livekit-starter
cd backend  && uv sync && uv run python src/agent.py download-files
cd ../frontend && pnpm install

2. Add your API keys — safely. Create .env.local in both backend/ and frontend/ (copy from each .env.example). You need LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, MURF_API_KEY, DEEPGRAM_API_KEY, and a GOOGLE_API_KEY.

Never commit keys. .env.local / .env.* are gitignored (with !.env.example as the one exception). Keys go in the deployment platform's secret store, never in code, never in a screenshot. Before you push, check your diff.

3. Run it:

# Terminal 1 — backend agent
cd backend && uv run python src/agent.py dev
# Terminal 2 — frontend
cd frontend && pnpm dev

4. Connect and test: open http://localhost:3000, click Start talking, allow the mic, and talk. Want to test with no UI at all? uv run python src/agent.py console gives you a terminal-only conversation.

5. Make it yours: everything the agent is lives in SYSTEM_PROMPT in backend/src/agent.py. Change that string and you change the whole agent — support bot, tutor, receptionist. Change the voice in murf.TTS(...) to pick from Murf's voice library. Add a capability by writing a method with the @function_tool decorator.

What I'd improve next

• Real scheme data. Eligibility currently comes from a curated local dataset; wiring it to an official, dated source would make the numbers authoritative.
• Voice authentication for memory. Today, memory is keyed on a spoken name. For anything more sensitive, that's too weak — I'd want a proper caller identity.
• Latency metrics on the dashboard. I track call outcomes; I'd add time-to-first-audio and turn latency so I can prove the "it feels instant" claim with numbers.

The code and the details

• Repo: https://github.com/praveenraj027/murf-livekit-starter
• Murf Falcon: the fastest production TTS — ~55ms model latency, ~130ms time-to-first-audio — which is what makes a voice helpline feel like a conversation instead of a walkie-talkie.

If you're thinking about who your app doesn't reach today, voice is often the answer — and it's more approachable to build than you'd think. Pick a real person, give your agent a clear job and hard limits, and start talking to it.

Built for 10 Days of Voice Agents — VoiceForBharat Edition. #VoiceForBharat