mermaid
graph TD
    User([User Voice]) <-->|WebRTC / SIP| LT[LiveKit Real-time Transport]
    LT <-->|Audio Streams| Agent[Voice Agent Session]
    Agent -->|Speech-to-Text| STT[Deepgram Nova-3]
    Agent -->|LLM Reasoner| LLM[Gemini 1.5 Flash]
    Agent -->|Text-to-Speech| TTS[Murf Falcon API]
    LLM <-->|Tool Calling| DB[(SQLite Persistent Memory)]
    LLM <-->|Context Handoff| Specialist[Returns Specialist Agent]
3. Why Voice?
Text-based chat apps fail to capture the nuances of speech—pronunciation, hesitation, and cadence. Spoken interaction is essential for building verbal confidence. In India, where mobile internet is ubiquitous but literacy rates vary, voice is the most natural, accessible, and frictionless interface.

By utilizing Murf Falcon, a high-fidelity, low-latency Text-to-Speech (TTS) engine, the agent communicates in warm, authentic Indian accents (like the default hi-IN-kabir). This minimizes the intimidation barrier often associated with dry, robotic western accents.

4. How the System Works
The architecture utilizes a real-time media transport layer combined with decoupled cognitive and speech engines:

Frontend: A responsive glassmorphic dashboard (index.html) featuring real-time visualizers, call statistics, and interactive controls.
Real-time Transport: LiveKit WebRTC serves as the audio pipeline, handling sub-100ms bidirectional streaming.
Speech-to-Text: Deepgram Nova-3 translates Hinglish voice inputs to text.
Brain (LLM): Gemini 1.5 Flash manages dialogue flow, assesses learner level, and invokes database tools.
Text-to-Speech: Murf Falcon synthesizes natural, speech-tuned Hinglish responses.
Data & State Layer: A local SQLite database (shiksha_vani_memory.db) storing learner profiles, call analytics, and human escalation tickets.
5. Key Features
Indian Accents & Code-Mixing: Supports Hinglish code-mixing. If a learner says "Mujhe English grammar seekhna hai," the agent replies in a corresponding encouraging tone: "Bohot achha! Hinglish me baat karke English seekhna easy ho jata hai. Let's practice!"
SQLite Memory & Consent-Before-Save: The agent checks if the user has spoken before (lookup_learner). It remembers names and previous topics but strictly requests consent before saving updates ("क्या मैं यह याद रख लूँ अगली बार के लिए?").
API Tool Chaining: Integrates a live dictionary and weather service. The agent checks the learner's district (e.g., Patna), fetches live weather via Open-Meteo, and creates a localized translation challenge ("The weather in Patna is rainy and 29 degrees today. Can you translate this to Hindi?").
Telephony & Outbound Retries: Initiates automated practice calls via LiveKit SIP. If the callee is busy (SIP 486/603) or doesn't answer (SIP 408/480), the agent logs the outcome and schedules retries in SQLite.
Human-in-the-Loop & Consent Escalation: If a learner expresses distress ("English is too hard, I want to quit"), the agent requests permission to alert a human teacher, redacts any sensitive PII (like credit cards, PINs, or bank details), and files a ticket with a date-based Reference ID (HF-YYYYMMDD-XXX).
Multi-Agent Handoff: If a user shifts from practicing English to asking about order returns from a sponsor store, the agent cleanly transfers the call to a CommerceSpecialistAgent using LiveKit's dynamic agent update method, returning them when done.
6. The Hard Parts
Challenge 1: Windows Path Length Limits on External Packages
Problem: While installing Twilio and LiveKit dependencies inside the Python environment, the build failed due to the Windows 260-character file path limit.
Investigation: Long nested namespaces in the dependencies exceeded Windows' default path thresholds.
Solution: Moved dependencies to a local self-contained libs directory under backend/src/libs and appended it to the system path at runtime using:
python


current_dir = os.path.dirname(os.path.abspath(__file__))
libs_path = os.path.join(current_dir, "backend", "src", "libs")
if libs_path not in sys.path:
    sys.path.insert(0, libs_path)
Lesson: When deploying voice agents on Windows, always budget for path limit workarounds or containerize your backend early.
Challenge 2: SSL Certification Errors on Public API Calls
Problem: In the middle of dictionary and weather tool calling, Python's urllib crashed with SSL: CERTIFICATE_VERIFY_FAILED errors on Windows.
Investigation: Python on Windows does not automatically use the OS's root certificates, causing SSL handshakes with public endpoints (api.dictionaryapi.dev) to fail.
Solution: Configured a custom SSL context bypass directly in the lookup function:
python


import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
# Pass ctx to urllib.request.urlopen(req, context=ctx)
Lesson: External tool calls must fail gracefully. Bypassing validation (in dev) or bundling certifi (in prod) is essential to avoid blocking the voice loop.
7. How You Can Build One
To build your own agent, follow the real-time audio pipeline flow:

Transport: Set up a WebRTC session using LiveKit.
STT: Capture user input and transcribe it with a model that supports multi-lingual/code-mixed audio.
LLM: Build system prompts that enforce short, voice-friendly replies (max 1–2 sentences) and structure your functions as tools.
TTS: Connect to Murf Falcon to generate fast, low-latency audio chunks.
Environment Setup
Create a .env.local file in the root directory:

env


MURF_API_KEY=your_murf_api_key_here
LIVEKIT_URL=your_livekit_url_here
LIVEKIT_API_KEY=your_livekit_api_key_here
LIVEKIT_API_SECRET=your_livekit_api_secret_here
# Optional Telephony Config
SIP_TRUNK_ID=your_sip_trunk_id
TWILIO_ACCOUNT_SID=your_twilio_sid
TWILIO_AUTH_TOKEN=your_twilio_token
IMPORTANT

Never commit your .env.local or .env files to git. Add them to your .gitignore to prevent exposing API keys.

8. Running the Project
Clone the repository:
bash


git clone [YOUR_GITHUB_REPOSITORY_URL]
cd voice-for-bharat-challenge-2026
Install dependencies:
bash


pip install -r requirements.txt
Run the web simulator (REST & Web interface):
bash


python web_demo.py
Run the LiveKit Voice Agent pipeline:
bash


python backend/src/agent.py dev
9. Testing the Agent
Open http://localhost:8085 in your browser.
Select a voice accent (e.g., "Kabir").
Click Connect and allow microphone access.
Test Case 1 (Hinglish Greeting): Say "Hello, mera naam Rohan hai."
Expected behavior: The agent greets you by name and asks if it can save your profile. Click Accept or say "Yes".
Test Case 2 (Tool Calling): Say "Define perseverance."
Expected behavior: The agent returns the definition, and a dictionary card slides down on the screen.
Test Case 3 (Guardrail Refusal): Say "Write a 500-word essay for my homework."
Expected behavior: The agent refuses to write the essay, and guides you to construct sentences yourself.
10. What I Would Improve Next
On-Device VAD Tuning: Fine-tune Silero VAD parameters to better handle ambient background noise in loud classroom settings.
Offline/On-Device TTS Fallback: Cache common voice prompts locally to ensure standard greetings can play instantly even during internet drops.
11. Repository and Demo
Code Repository: [YOUR_GITHUB_REPOSITORY_URL]
Live Web Demo: [YOUR_DEMO_URL] (if applicable)
12. What I Learned
Building Shiksha Vani taught me that engineering a voice agent is vastly different from building a chatbot. Turn-taking, speech pacing, VAD tuning, and TTS latency (~55ms with Murf Falcon) represent the difference between a natural human conversation and a disjointed, frustrating interaction. Designing with guardrails, strict consent checks, and robust fallbacks is crucial for building systems that Indian learners can trust.





![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/xijo98gkep6qtr51z48s.png)