What started as a simple voice agent became something much bigger over the last ten days.
I built MoneyBuddy, a multilingual AI voice agent for the Financial Services track of the 10 Days of Voice Agents — VoiceForBharat Edition.
The goal was simple:
Make financial guidance easier to access through a natural voice conversation.
Instead of forcing users to navigate complicated forms, search through scheme websites, or understand financial terminology on their own, MoneyBuddy lets them simply talk.
Over ten days, it evolved from a basic voice conversation into a system with persistent memory, financial tools, outbound calling, human escalation, analytics, and specialist-agent handoffs.
And the biggest lesson was that building a useful voice agent is much more than connecting an LLM to a microphone.
The Problem I Wanted to Solve
Government financial schemes can provide meaningful support, but discovering whether a scheme is relevant, understanding eligibility, finding required documents, and knowing what to do next can be difficult.
For many users, especially people who are more comfortable speaking than typing, voice can make that interaction much more natural.
That became the idea behind MoneyBuddy:
A voice-first financial companion that explains, guides, and knows when it should stop and ask for help.
MoneyBuddy focuses on:
- Government-scheme guidance
- Financial literacy
- Eligibility information
- Document guidance
- Fraud awareness
It is intentionally not a banking transaction system.
MoneyBuddy does not ask users for OTPs, PINs, passwords, CVVs, full card numbers, or sensitive banking credentials.
What MoneyBuddy Became
The project started with a simple voice loop:
User speaks → AI understands → AI responds
By the end of the challenge, it had grown into a much larger system.
MoneyBuddy can now:
- Speak using an Indian voice powered by Murf Falcon
- Understand English, Hindi, and Hinglish
- Follow financial safety guardrails
- Show real-time agent states in the frontend
- Display a live transcript
- Remember returning callers using SQLite
- Retrieve memory through function tools
- Retrieve government-scheme information through a domain tool
- Chain stored caller information into financial lookups
- Make outbound phone calls
- Ask permission before human escalation
- Create escalation requests
- Track call outcomes through analytics
- Hand conversations to a Government Scheme Specialist
- Return conversations from the specialist to the main agent
The important shift was this:
A voice agent becomes a real application when you add state, tools, memory, safety, observability, and failure handling around the conversation loop.
How MoneyBuddy Works
At the core, MoneyBuddy combines four major components:
- Deepgram STT — converts speech into text.
- Gemini LLM — handles reasoning, conversation, routing, and tool selection.
- Murf Falcon TTS — converts generated responses back into speech.
- LiveKit — handles real-time audio transport.
For outbound phone calls, LiveKit SIP connects the voice agent to telephony.
The complete architecture
flowchart LR
U["👤 User<br/>Browser / Phone"] --> LK["LiveKit<br/>Real-time Audio"]
LK --> STT["Deepgram STT<br/>Speech → Text"]
STT --> LLM["Gemini LLM<br/>Reasoning + Routing"]
LLM <--> MEM["SQLite<br/>Caller Memory"]
LLM <--> TOOL["Financial Tools<br/>Scheme Data"]
LLM --> SPEC["Government Scheme<br/>Specialist Agent"]
MEM --> LLM
TOOL --> LLM
SPEC --> LLM
LLM --> TTS["Murf Falcon<br/>Text → Speech"]
TTS --> LK
LK --> U
This separation is important.
The LLM is not responsible for everything.
- Memory is handled through tools.
- Scheme information comes from grounded data.
- Human escalation is handled through a dedicated tool.
- Specialist routing is handled separately.
- LiveKit manages real-time communication.
That made the system easier to reason about and safer to extend.
📸 MoneyBuddy Frontend
The frontend was redesigned around the actual states of a voice conversation.
Instead of displaying only a microphone button, MoneyBuddy clearly communicates whether it is:
- Ready
- Connecting
- Listening
- Speaking
- Call ended
It also provides:
- Live transcript
- Microphone permission guidance
- Language selection
- Financial safety messaging
- Reconnection/loading states
- Call-ended controls
The interface is intentionally simple.
The user should understand what to do without knowing anything about LiveKit, STT, LLMs, or TTS.
1. Giving MoneyBuddy an Indian Voice
The first challenge was getting a voice agent working end-to-end.
I chose an Indian voice because the target experience is designed for Indian users.
MoneyBuddy uses Murf Falcon for text-to-speech.
Voice quality matters especially for a financial assistant.
A system that gives correct information but sounds robotic, unnatural, or difficult to understand quickly loses trust.
Murf Falcon became an important part of the project because voice-agent UX is heavily affected by the time between:
User stops speaking → Agent starts speaking
That made latency a product decision, not just an engineering metric.
2. Personality, Guardrails, and Multilingual Conversations
Once the basic voice loop worked, MoneyBuddy needed a clear job and strict boundaries.
I defined three primary objectives:
- Government-scheme guidance
- Financial literacy
- Fraud awareness
Then I added financial safety rules.
MoneyBuddy must never request:
- OTPs
- PINs
- Passwords
- CVVs
- Full card numbers
- Sensitive banking credentials
It also cannot promise guaranteed scheme approval or financial outcomes.
Voice-first prompting
A response that looks good in a chat window can sound terrible when spoken.
For voice, I had to think differently.
MoneyBuddy uses:
- Short sentences
- Natural phrasing
- Simple vocabulary
- Clear pauses
- No raw JSON
- No markdown-style responses
- No unnecessary technical terminology
I also added support for Hindi and Hinglish.
The goal was not to make users adapt to the system.
The system should adapt to the way users naturally communicate.
3. Building a Frontend for Voice
The frontend became more than a microphone interface.
It visually represents the current state of the agent:
Ready → Connecting → Listening → Speaking → Call Ended
The live transcript makes it clear who said what.
Microphone permission errors are also handled explicitly instead of leaving the user wondering why nothing is happening.
For a voice product, this feedback is essential.
When there is no visible state, a few seconds of network delay can feel like the entire application has crashed.
4. Giving MoneyBuddy Memory
A returning caller should not have to start from zero every time.
I added persistent SQLite memory.
A caller record can contain:
user_id
name
language_preference
facts
last_interaction
But there was an important architectural decision here.
I did not want to dump the entire database into the system prompt.
Instead, MoneyBuddy has function tools for memory operations.
For example:
@function_tool
async def lookup_caller(user_id: str):
"""Look up a caller's saved profile and relevant financial facts."""
return get_caller(user_id)
The model decides when it needs caller information and calls the function.
This creates a much cleaner separation:
Application data → Tool → LLM
rather than:
Entire database → Prompt → LLM
I also added sanitization before storing financial information so sensitive credentials are not persisted as normal caller facts.
Most importantly:
MoneyBuddy asks for permission before saving information.
5. Giving the Agent Real Financial Data
Memory alone isn't enough.
The agent also needs reliable domain information.
For Day 5, I added a grounded local dataset containing Indian government financial schemes such as:
- PM Kisan Samman Nidhi
- PM Suraksha Bima Yojana
- PM Jeevan Jyoti Bima Yojana
- Atal Pension Yojana
- PM Mudra Yojana
MoneyBuddy accesses this information through a function tool.
Instead of relying entirely on the LLM's internal knowledge, the agent can retrieve structured scheme information when required.
The dataset also contains recency information so the agent can communicate when its information was last updated.
The principle was simple:
When an answer depends on structured domain data, use a tool instead of hoping the LLM remembers the correct answer.
6. Making MoneyBuddy Call Users
On Day 6, MoneyBuddy stopped waiting for users to initiate every conversation.
I added outbound calling through LiveKit SIP.
The use case was:
A reminder for someone already identified as eligible for a government scheme with an approaching deadline.
Outbound calling requires a different conversational design.
The user did not ask for the call.
So the opening needs to establish three things immediately:
Who is calling.
Why they are calling.
How the user can stop the call.
That makes the interaction more transparent and respectful.
Outbound interaction
flowchart TD
A["📞 Outbound Call Started"] --> B["User Answers"]
B --> C["MoneyBuddy identifies itself"]
C --> D["Explains reason for calling"]
D --> E["Provides opt-out"]
E --> F{"User response"}
F -->|Continue| G["Deliver scheme reminder"]
F -->|No / Stop| H["End call respectfully"]
B -->|No answer| I["Call outcome recorded"]
B -->|Busy| I
B -->|Immediate hang-up| I
One limitation remained here: carrier-level retry handling for busy, no-answer, and voicemail outcomes was not implemented as a custom retry queue.
The challenge was primarily about demonstrating the outbound interaction itself, so I kept the implementation focused rather than adding an unnecessary retry infrastructure.
7. Knowing When AI Should Ask a Human
One of the biggest lessons from this challenge was:
A good AI agent should know when it should stop.
For MoneyBuddy, human escalation is triggered for situations such as:
- Possible financial fraud
- Problems requiring a decision the AI cannot make
The agent prepares a short summary containing only useful information:
- What happened
- What the agent already checked
- Urgency
- Preferred language
- Preferred follow-up method
Before sharing that information, MoneyBuddy asks the caller for permission.
Sensitive credentials such as OTPs, PINs, passwords, and account numbers are excluded.
The caller receives a reference ID and an explanation of what happens next.
Human escalation flow
flowchart LR
U["User"] --> M["MoneyBuddy"]
M --> Q{"Needs human help?"}
Q -->|No| C["Continue conversation"]
Q -->|Yes| S["Explain what will be shared"]
S --> P{"Caller gives permission?"}
P -->|No| N["Do not create request<br/>Continue safely"]
P -->|Yes| E["Create Escalation"]
E --> R["Reference ID"]
R --> X["Explain next step"]
This was an important safety boundary.
The AI doesn't silently send the user's information somewhere just because it thinks escalation is useful.
8. Measuring the System Instead of Guessing
A voice application needs observability.
As part of the challenge, I built a call analytics layer that records call outcomes and exposes them through a dashboard.
The dashboard is designed to track:
- Total calls
- Successful calls
- Failed calls
- Success rate
- Call history
- Failure categories
- Track-specific outcomes
- Latency
Rather than presenting test-suite results as real-world call statistics, I kept the distinction clear.
The implementation was validated through automated testing across the different challenge features.
By Day 9, the specialist routing tests passed:
14/14 tests passed
The broader behavioral test suite passed:
16/16 tests passed
And the backend pytest suite passed:
4/4 tests passed
These results demonstrate that the core workflows were functioning as implemented, while the dashboard provides the foundation for collecting real call metrics as the system is used.
This distinction matters.
Automated tests tell me whether the system behaves correctly under defined scenarios. Real call metrics tell me how the system performs with actual users.
For this challenge, I chose not to invent latency or call-success numbers that I could not independently verify.
9. The Multi-Agent Step
The final major technical step was turning MoneyBuddy into a multi-agent system.
The main agent remains the general financial assistant.
When the user needs deeper government-scheme knowledge, MoneyBuddy can hand the conversation to a dedicated:
Government Scheme Specialist
The specialist has its own role, instructions, and boundaries.
The user should not have to explain the entire problem again.
Specialist handoff architecture
flowchart TD
U["👤 User"] --> M["MoneyBuddy"]
M --> Q{"Does the request<br/>need specialist knowledge?"}
Q -->|No| M
Q -->|Yes| A["Announce handoff"]
A --> H["Handoff Tool"]
H --> S["Government Scheme<br/>Specialist"]
S --> C["Continue same conversation"]
C --> D{"Task complete?"}
D -->|No| S
D -->|Yes| R["Return to MoneyBuddy"]
D -->|Topic changed| R
R --> M
This is one of the places where a multi-agent architecture becomes useful.
Instead of giving one giant prompt to one giant agent, each agent gets a narrower responsibility.
10. Solving Handoff Latency
The specialist handoff introduced another voice-specific challenge.
If the system waited for the specialist's entire LLM response before speaking, the user could experience an awkward silence.
So I changed the flow.
The specialist can immediately acknowledge the handoff:
await session.say(
"Hi, I'm the Government Scheme Specialist. "
"I'll help you with that."
)
Then the detailed response can continue generating.
The goal is to separate:
"The specialist has taken over."
from:
"The specialist has finished generating the complete answer."
That small architectural change makes the handoff feel much faster.
11. Deterministic Agent Identity
Another interesting problem appeared in the frontend.
Initially, the UI could try to determine which agent was active by inspecting transcript text.
That is unreliable.
For example, if the normal MoneyBuddy agent happened to say the word "specialist", the frontend could incorrectly display the specialist state.
So I changed the architecture to use explicit application state / LiveKit participant metadata.
Conceptually:
active_agent = "moneybuddy"
or:
active_agent = "specialist"
The frontend reads the state directly.
This led to another useful lesson:
UI state should come from application state, not guesses extracted from conversation text.
12. The Design Decision I Changed
One of the most important design decisions I changed during the challenge was how MoneyBuddy handled memory.
The initial temptation was to load stored caller information directly into the system prompt.
It works, but it creates problems.
The prompt becomes larger.
The model receives information it may not need.
And the separation between application data and model instructions becomes weaker.
So I changed the architecture to:
Caller needs information
↓
Memory lookup tool
↓
SQLite
↓
Relevant data returned
↓
LLM uses only what it needs
This made the memory architecture cleaner and easier to control.
13. The Hard Parts
The hardest part wasn't getting an LLM to answer questions.
It was making all the components behave reliably together as a real-time voice application.
Multilingual voice handling
Hindi and Hinglish required more than simply changing the prompt.
STT, language detection, LLM output, writing script, and TTS configuration all had to work together.
Voice formatting
Text that looks fine on a screen can sound terrible when spoken.
I had to continuously simplify responses, remove formatting artifacts, and keep sentences short.
Handoff latency
Switching agents introduced another potential delay.
The solution was to immediately speak the specialist introduction rather than waiting for the complete response.
Agent identity
Transcript-based UI detection caused unreliable specialist labels.
Explicit state solved that problem.
Keeping previous days stable
Every new feature had to coexist with everything already built.
I maintained separate day branches and repeatedly ran the existing test suites before pushing changes.
This helped prevent later features from accidentally breaking earlier functionality such as memory, analytics, or guardrails.
14. Testing and Reliability
One thing I deliberately focused on throughout the challenge was not just adding features, but verifying that existing features remained intact.
By Day 9, the project had dedicated tests covering:
- Behavioral agent behavior
- Memory functionality
- Financial guardrails
- Day 8 analytics
- Day 9 specialist routing
- Specialist handoff behavior
- Fallback handling
The final Day 9 routing test suite passed:
14/14 tests
The broader behavioral suite passed:
16/16 tests
And the backend pytest suite passed:
4/4 tests
These are automated test results, not real-user success metrics.
That distinction is important.
15. Running MoneyBuddy Yourself
The complete Day 10 repository is available here:
https://github.com/tanush326k/murf-livekit-starter/tree/day10
The project contains a backend voice agent and a Next.js frontend.
The main architecture looks roughly like:
murf-livekit-starter/
│
├── backend/
│ ├── src/
│ │ ├── agent.py
│ │ ├── prompt.py
│ │ ├── db.py
│ │ ├── schemes_data.json
│ │ └── outbound.py
│ │
│ └── tests/
│
└── frontend/
├── components/
├── app/
└── styles/
Main components
The project uses:
- Python
- LiveKit
- Deepgram
- Gemini
- Murf Falcon
- SQLite
- Next.js
Environment variables
Create your local environment configuration and add your own credentials.
For example:
MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
GOOGLE_API_KEY=your_key_here
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
Never commit real API keys to GitHub.
After installing the backend and frontend dependencies, start the required services, open the frontend, allow microphone access, and start a conversation.
The exact setup commands are documented in the repository.
16. Troubleshooting
The agent connects but doesn't speak
Check the backend logs first.
Then verify:
- LiveKit credentials
- Deepgram configuration
- Gemini configuration
- Murf credentials
- TTS voice configuration
A voice agent is a pipeline.
One broken component can make the entire application appear silent.
Hindi is understood but sounds wrong
Check both sides of the pipeline.
The STT configuration must support multilingual input.
The TTS configuration must use the appropriate Indian voice configuration.
The LLM should also return Hindi using Devanagari when Hindi output is expected.
The specialist handoff feels slow
Don't wait for the entire specialist response before starting audio.
Give the specialist an immediate spoken introduction and then continue generating the detailed response.
The frontend shows the wrong active agent
Don't infer agent identity from transcript text.
Use explicit application state or LiveKit participant metadata.
17. What I Would Improve Next
If I continued building MoneyBuddy beyond the challenge, I would focus on production readiness.
The next areas would include:
- Stronger multilingual evaluation
- More comprehensive government-scheme sources
- More robust telephony retry handling
- Better tool observability
- Stronger authentication and privacy controls
- Better human-support workflows
- More detailed latency breakdowns
- Production-grade monitoring
- Larger-scale testing with real conversations
The goal would be to move from a challenge project toward a system that could be responsibly used in a real financial-support environment.
What Ten Days Taught Me
The biggest lesson from this challenge is that building a voice agent isn't primarily about choosing the biggest model.
It's about everything around the model.
You need:
Good speech recognition.
Fast text-to-speech.
Clear instructions.
Strong guardrails.
Useful tools.
Persistent state.
Reliable transport.
Observability.
Human escalation.
Graceful failure handling.
The LLM is only one part of the system.
Once voice becomes the interface, latency, state, turn-taking, and response formatting become product decisions — not just engineering details.
MoneyBuddy started as an agent that could hear me and talk back.
Ten days later, it can remember users, access financial data, make outbound calls, escalate difficult situations, measure conversations, and hand complex questions to a specialist.
That progression was the most valuable part of the challenge.
Final Takeaway
If you're building your own voice agent, my biggest advice is:
Don't start by trying to make it do everything.
Make it talk.
Then give it a job.
Then give it boundaries.
Then give it tools.
Then give it memory.
Then make it reliable.
And only after that, start making it smarter.
Project
MoneyBuddy — Day 10 Repository
https://github.com/tanush326k/murf-livekit-starter/tree/day10
Challenge:
10 Days of Voice Agents — VoiceForBharat Edition
Built with:
LiveKit • Deepgram • Gemini • Murf Falcon • Python • Next.js • SQLite
If this project helps you build your own voice agent, I'd love to see what you create.
