🌾 1. The Problem & The Mission
Across rural India, millions of farmers rely on timely agricultural information—such as sowing advisories, weather forecasts, pest outbreak alerts, and Mandi (market) crop rates—to make daily decisions that affect their livelihoods. However, traditional text-based mobile apps fail for rural populations due to language barriers, literacy constraints, and complex navigation UI.

Voice is the natural interface for Bharat.

During the 10 Days of Voice Agents — #VoiceForBharat Edition challenge, we set out to build KisanVani (किसान वाणी): an empathetic, expert AI voice assistant and digital Krishi Mitra (Agriculture Friend). Built specifically for the Farm & Field track, KisanVani allows farmers to speak naturally in Hindi, Hinglish, or Indian English, receiving real-time agricultural guidance, market rates, and weather alerts with human-like responsiveness.

🏗️ 2. System Architecture & Core Stack
Building a real-time voice agent requires tightly integrated speech processing, language modeling, dynamic tool invocation, and streaming synthesis. Audio latency must remain under 300ms so conversations feel natural without awkward delays.

Mermaid diagram
The Technology Stack:
Speech-to-Text (STT): Deepgram Nova-3 (language="multi") for robust multilingual recognition.
Text-to-Speech (TTS): Murf Falcon 2 (livekit-murf) — the fastest TTS API on the market, streaming audio in under 200ms with natural Indian English and Hindi pronunciation (Anisha / Easha voices).
Large Language Model (LLM): Google Gemini 2.5 Flash for high-speed reasoning, function calling, and multilingual fluency.
Real-time Transport & Pipeline: LiveKit Agents SDK (livekit-agents ~1.4) with Silero VAD and Multilingual Turn Detection.
Persistent Memory & Storage: SQLite database for privacy-compliant caller profiles, outbound telephony logs, and human escalation tickets.
⭐ 3. Important Features Built
Over 10 days of iterative development, KisanVani evolved from a basic voice pipeline into a comprehensive production-grade AI agent system:

1. Ultra-Low Latency Indian Voice (Murf Falcon 2)
   Using Murf Falcon 2 TTS with sentence-level tokenization (SentenceTokenizer(min_sentence_len=2)), KisanVani achieves sub-250ms audio synthesis. The voice sounds empathetic, warm, and natural—essential for building trust with farming communities.

2. Personality, Objectives & Safety Guardrails
   KisanVani operates under strict system prompt guardrails:

Scope Control: Hard refusals for non-agricultural, medical, legal, or financial requests.
Safety Net: Out-of-scope or unverified queries are gracefully directed to the official Kisan Toll-Free Helpline (1800-180-1551).
Zero Hallucination: If tool data is unavailable, the agent speaks a polite fallback out loud rather than inventing prices or weather numbers.

1. Multilingual Support & Native Script Enforcement
   The agent recognizes Hindi, Hinglish, and English. A strict rule enforces native script output for non-English languages (Devanagari script for Hindi, e.g., "नमस्ते", never romanized "namaste"), ensuring Murf Falcon synthesizes native phonemes cleanly.

2. Opt-in Caller Memory & Privacy Protocols (SQLite)
   Returning callers are greeted warmly by name and past farm context (e.g., land size, crops grown, home district). Memory is bound by a Hard Privacy Consent Rule: the agent must explicitly ask permission before saving facts into SQLite (save_caller_memory) and supports a full "Forget Me" wipe protocol (forget_caller_memory).

3. Real-Time Agricultural Tools & Tool Chaining
   get_weather_forecast: Queries Open-Meteo REST API for live temperature, humidity, rain probability, and spraying suitability advice.
   get_mandi_prices: Retrieves current market modal rates and min/max price ranges per quintal across wheat, paddy, mustard, cotton, potato, and onion markets.
   Tool Chaining: If a farmer asks "What is the weather today?" without mentioning their location, KisanVani automatically checks saved caller memory for their home district.

4. Proactive Outbound Alerts & Telephony Protocols
   KisanVani can initiate outbound SIP calls for emergency weather or pest warnings. Outbound calls follow a strict 3-step opening protocol:

Who is calling: Identify as KisanVani AI Krishi Mitra.
Why calling: State the specific alert reason.
How to stop: Explain how to opt out (opt_out_alerts).

1. Human Escalation Protocol with Reference Tickets
   For severe crop blight epidemics or complex subsidy disputes beyond AI scope, KisanVani creates a human escalation ticket via create_escalation. After obtaining caller consent, it logs a sanitized record in SQLite, issues a reference ID (e.g. ESC-48291), and promises a 24-hour callback from a senior Krishi Officer.

2. Multi-Agent Handoff (Assistant ↔ CropSpecialist)
   When questions turn to complex crop pathology, yellow rust, or pesticide dosage, the main assistant dynamically transfers the session to CropSpecialist (Fasal Visheshagya). If the caller later asks about weather or Mandi rates, the specialist hands control seamlessly back to the main agent.

💻 4. Code Snippets & Walkthrough
Here is how the core pipeline and dynamic handoff are configured in Python (backend/src/agent.py):

LiveKit Pipeline & Murf Falcon Configuration
python

from livekit.plugins import deepgram, google, murf, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel
from livekit.agents import AgentSession, tokenize

Voice AI pipeline using Murf Falcon (Anisha voice), Gemini LLM, and Deepgram Nova-3

session = AgentSession(
stt=deepgram.STT(model="nova-3", language="multi"),
llm=google.LLM(model="gemini-2.5-flash"),
tts=murf.TTS(
voice="Anisha",
style="Conversation",
tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),
text_pacing=True,
),
turn_detection=MultilingualModel(),
vad=silero.VAD.load(),
preemptive_generation=True,
)
Real-Time Weather Tool with Memory Chaining & Graceful Fallback
python

@function_tool
async def get_weather_forecast(self, ctx: RunContext, district: str = "") -> str:
"""Fetch live weather forecast and agricultural spraying advice."""
target_district = district.strip()

# Tool Chaining: Fallback to saved caller memory if district not provided
if not target_district:
    record = get_caller(self.user_id)
    if record and record.get("facts", {}).get("district"):
        target_district = record["facts"]["district"]
    else:
        target_district = "Karnal"
try:
    # Fetch geocoding and live Open-Meteo forecast data...
    # Returns temperature, humidity, rain probability, and spraying advice
    return f"डेटा दिनांक {today_str} के अनुसार: {place} में तापमान {temp}°C, वर्षा की संभावना {precip_prob}% है।"
except Exception:
    # Graceful spoken fallback
    return "मौसम सेवा API से संपर्क विफल रहा। कृपया किसान हेल्पलाइन 1800-180-1551 पर कॉल करें।"

Dynamic Multi-Agent Specialist Handoff
python

@function_tool
async def transfer_to_crop_specialist(self, ctx: RunContext, issue_description: str = "") -> str:
"""Transfer session to specialized Crop Disease & Pest Specialist."""
specialist = CropSpecialist(user_id=self.user_id)
ctx.session.update_agent(specialist)
return "मैं आपको हमारे फ़सल रोग और कीट विशेषज्ञ से कनेक्ट कर रहा हूँ। कृपया एक क्षण प्रतीक्षा करें।"
⚡ 5. Real Engineering Challenges & Solutions
Building KisanVani wasn't without hurdles. Here are three key technical challenges we encountered and resolved:

Challenge 1: Script Normalization & TTS Mispronunciations
Problem: Romanized Hindi (Hinglish like "Namaste, aapka swagat hai") caused Murf Falcon to read Hindi words with an English accent, reducing audio naturalness.
Solution: Implemented prompt-level native script enforcement. All Hindi outputs are strictly rendered in Devanagari script ("नमस्ते, आपका स्वागत है"). Murf Falcon's phonetic parser handles Devanagari flawlessly, producing authentic Indian accents.
Challenge 2: Turn Detection in Outdoor Rural Environments
Problem: Standard Voice Activity Detection (VAD) models falsely triggered background noise (tractor engine noise, wind, animal sounds) or cut off farmers during natural pauses in speech.
Solution: Combined Silero VAD with LiveKit's MultilingualModel turn detector and enabled Deepgram telephony noise cancellation filters (noise_cancellation.BVC()). This allowed the agent to wait for true completion of user speech while suppressing background outdoor noise.
Challenge 3: Context Preservation During Dynamic Agent Handoffs
Problem: Updating the active agent (ctx.session.update_agent(specialist)) risked losing caller state and background memory mid-conversation.
Solution: Passed the active user_id context directly into the initialized CropSpecialist instance, enabling the specialist to query SQLite memory and maintain seamless context across agent transfers.
🚀 6. How to Build & Run KisanVani
You can easily clone, build, and run KisanVani locally!

Prerequisites
Python 3.10+ with uv installed (pip install uv)
Node.js 18+ and pnpm
API Keys for: LiveKit Cloud, Murf AI, Deepgram, and Google Gemini
Step 1: Clone the Repository
bash

git clone https://github.com/Dharmesh-jagatiya/VoiceAgentOfBharat.git
cd VoiceAgentOfBharat
Step 2: Configure Environment Variables
Copy .env.example to backend/.env.local and add your secret API keys:

env

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_gemini_api_key
And in frontend/.env.local:

env

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_secret
AGENT_NAME=kisanvani-farm-agent
Step 3: Run the Backend Agent
bash

cd backend
uv sync
uv run python src/agent.py dev
Step 4: Run the Frontend UI
In a separate terminal:

bash

cd frontend
pnpm install
pnpm dev
Open http://localhost:3000 in your browser, click "Connect to KisanVani AI", and start speaking!

🔮 7. Future Roadmap & What's Next
Multilingual Regional Dialects: Support for Gujarati, Punjabi, Marathi, and Kannada audio rendering.
Multimodal Plant Pathology: Allowing farmers to capture photos of diseased crop leaves via smartphone camera while talking to KisanVani for visual AI diagnosis.
Offline SMS Alert Fallback: Sending SMS reference receipts automatically after every voice escalation or Mandi query.
🔗 8. Project Links & References
GitHub Repository: Dharmesh-jagatiya/VoiceAgentOfBharat
Murf Falcon TTS Docs: murf.ai/api/docs/text-to-speech-models/falcon-2
LiveKit Agents Documentation: docs.livekit.io/agents
Challenge: #VoiceForBharat 10 Days of AI Voice Agents by Murf AI
Thank you to Murf AI and LiveKit for organizing the 10 Days of Voice Agents challenge! 🌾⚡