Building voice AI sounds deceptively clean on paper: capture speech, stream it to an STT engine, prompt an LLM, and synthesize audio back in real time.
In reality, building a real-time, multilingual voice assistant that handles high-stress banking fraud while running on desktop hardware in 30°C heat with unstable power is a completely different story.
As a Class 12 student diving into my first-ever hackathon, I spent the last 10 days of the #VoiceForBharat Challenge building Raksha—an empathetic, multi-agent voice assistant designed to protect Indian citizens from cyber scams, verify official government schemes, and escalate active financial fraud to human coordinators.
Here is the honest breakdown of how Raksha evolved from Day 1 to Day 10, the absurd hurdles along the way, and what it actually takes to build reliable voice systems.
1. The Core Problem: Panic & Digital Banking Scams
India's digital payments ecosystem is scaling at lightspeed, but so is financial fraud. Every day, people receive panic-inducing SMS messages claiming: "Your bank account is blocked. Update KYC immediately or share OTP."
When someone is panicked, reading a 5-page PDF advisory is the last thing they will do. They need immediate, reassuring spoken advice in accessible Hinglish: "Ghabraiye mat. Bank kabhi phone ya message par OTP nahi mangta. Yeh poori tarah fraud hai."
Voice is the most natural medium for this. If an AI agent can step in during those critical first 60 seconds, it can prevent immediate financial loss.
2. The 10-Day Journey: Power Cuts, Audio Hacks & Architecture
Days 1–3: The Rocky Start & The Audio Hack
Day 1 was pure chaos. I was battling 404s and gRPC hanging issues while connecting Gemini and LiveKit. Exactly 16 minutes before the midnight deadline, the power cut out completely. When it returned a minute past midnight, my desktop greeted me with a blue SrtTrail.txt Windows repair screen—and then another storm knocked the grid out again.
I thought I was out on Day 1, but after seeing an extension on Discord, I locked back in for Day 2.
Being on a desktop without a dedicated studio mic—and honestly, being an introvert where speaking out loud repeatedly into a mic for demos felt unnatural—I found a workaround: I routed clean TTS audio from my phone directly into my earphone inline mic. It bypassed background fan noise, kept latency clean, and fed Deepgram crisp input. By Day 3, I had rebuilt the frontend with a custom slate-indigo theme, 5 distinct agent states, and LiveKit's audio-reactive WebGL aura visualizer.
Days 4–5: Persistent Memory, Guardrails & Tool Calling
A voice assistant can't just be an open chatbot; it needs strict boundaries. On Day 4, I implemented persistent caller memory using SQLite. If a returning caller connects, Raksha recognizes them contextually. But in banking, privacy is non-negotiable: Raksha is strictly instructed to ask for explicit consent before saving any facts.
On Day 5, I hooked up dynamic database tools like check_scheme_eligibility with spoken timestamping ("August 2026 ke nirdeshon ke anusaar..."), ensuring Raksha queries official SQLite records on the fly rather than hallucinating scheme details.
Days 6–7: Outbound SIP Telephony & Human Escalation
Day 6 was built in 30°C+ heat with low voltage, a barely spinning ceiling fan, and a constantly beeping UPS. I integrated LiveKit SIP Telephony with Linphone to place outbound reminder calls for PMJJBY renewals, implementing an instant opt-out tool whenever STT caught "band karo".
On Day 7, I added human-in-the-loop escalation. If a user reports active unauthorized deductions, Raksha stops acting alone. She asks: "Kya main aapki shikayat hamari human security team ko bhej sakti hoon?" Upon consent, a regex sanitizer strips 4–6 digit OTPs and 10–16 digit account numbers from the summary, dispatching a color-coded ticket (e.g., ESC-96155) directly to a Discord webhook desk.
Real-time Discord escalation desk with emergency color-coding and automated regex PII redaction.
Days 8–9: Real-Time Telemetry & The Multi-Agent Split
On Day 8, I built an auto-refreshing telemetry command center using Flask and Chart.js, tracking call resolution rates live in SQLite without logging any private conversation text.
Raksha Command Center: Telemetry dashboard tracking session resolution rates and pipeline diagnostics in real time.
On Day 9, I addressed the biggest architectural bottleneck: prompt overload. Cramming fraud prevention, escalation rules, and insurance checklists into one system prompt caused hallucinations. I split the system into two focused agents:
- Raksha (Main Agent): Handles scam triage, OTP warnings, and emergency escalation using Murf Falcon's Anisha voice.
- Samar (Scheme Specialist): A dedicated consultant who verifies guidelines for schemes like PMJJBY using Murf Falcon's Samar voice.
When a caller asks about insurance, Raksha initiates a tool-driven handoff (transfer_to_scheme_specialist), passing the conversation context across worker processes so the user never has to re-explain their situation.
3. System Architecture & Real-Time Pipeline
To keep the conversation natural, end-to-end latency must stay well under a second. Here is the WebRTC pipeline powering Raksha:
[ Citizen Speaks ] ──► (WebRTC Audio Stream) ──► Deepgram Nova-3 STT
│
▼
Gemini 3.5 Flash-Lite LLM
│
(Tool Dispatch)
┌─────────┴─────────┐
▼ ▼
SQLite Database Discord Webhook Desk
(Memory & Schemes) (Human Escalations)
│
▼
Murf Falcon TTS (Anisha / Samar)
│
[ Spoken Response ] ◄─────────────────────────┘
Live WebRTC consultation session with audio-reactive visualizer and real-time state synchronization.
The Tech Stack:
- TTS (Text-to-Speech): Murf Falcon API (Ultra-low latency Indian voices: Anisha for Raksha and Samar for the Scheme Specialist).
-
STT (Speech-to-Text): Deepgram Nova-3 (Configured with
language="multi"for code-mixed Hinglish). - LLM Engine: Google Gemini 3.5 Flash-Lite via Google AI Studio.
- Transport & Orchestration: LiveKit Agents Framework over WebRTC.
- Storage & Telemetry: Local SQLite for stateful memory, scheme rate cards, and resolution logs.
4. Key Engineering Lessons Learned
1. State Bleed in Multi-Agent Handoffs
When transferring conversation history between agents, the incoming specialist reads the shared transcript. If system prompts lack hard identity boundaries, the incoming agent can get confused by previous greetings in the chat log. Ensuring that handoff tools explicitly define the incoming agent's persona is critical to preventing identity bleed.
2. The Phonetic STT Trap with Acronyms
During a live recording, saying the acronym "PMJJBY" rapidly caused Deepgram to transcribe it phonetically as "PFJYWise". Because the SQLite tool expected an exact string match, it returned Scheme not found.
The Lesson: Voice AI systems cannot rely on rigid text matching. Prompts must encourage full spoken titles (e.g., "Pradhan Mantri Jeevan Jyoti Bima Yojana"), and backend tools need flexible alias mappings.
3. VAD and Conversational Timing
Tuning Voice Activity Detection (VAD) is a delicate balance. If turn detection is too sensitive, the agent interrupts natural pauses; if it is too slow, response latency feels sluggish. Pairing Silero VAD, Deepgram Nova-3, and Murf Falcon's fast streaming TTS delivered the low-latency responsiveness needed for real-time triage.
5. How to Run This Project Locally
1. Clone & Set Up Backend
git clone https://github.com/PrakhidhaChawdhury/voice-for-bharat-day1
cd backend
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -r requirements.txt
2. Configure Environment Keys
Create a .env.local file inside the backend/ directory:
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
DEEPGRAM_API_KEY=your_key
MURF_API_KEY=your_key
GOOGLE_API_KEY=your_key
DISCORD_WEBHOOK_URL=your_discord_webhook
3. Launch Agent & Frontend Client
# Terminal 1: Run Backend Agent
python src/agent.py dev
# Terminal 2: Run Frontend Client
cd ../frontend
npm install
npm run dev
Backend agent runner and LiveKit WebRTC session logs during live execution.
Open http://localhost:3000 in your browser and start a voice consultation.
6. What's Next?
This 10-day sprint proved that voice AI can be empathetic, secure, and privacy-conscious. Moving forward, the roadmap includes:
- Live Bank Endpoints: Connecting live financial verification APIs instead of local SQLite mock tables.
- Vector RAG: Parsing dynamic policy PDFs for complex scheme queries.
- Direct SIP Trunking: Enabling toll-free inbound numbers so citizens on basic feature phones can call in without an internet connection.
Built as part of the 10 Days of Voice Agents Challenge (#VoiceForBharat), powered by Murf AI and LiveKit.
