From zero to a full-stack, multilingual agricultural voice agent with caller memory, real-time mandi tools, outbound price alert calls, human escalation, and specialist agent handoffs — powered by Murf Falcon & LiveKit.
🌟 The Problem & The Mission
In rural India, millions of farmers make critical livelihood decisions every day: When should I harvest? Will it rain before I spray pesticides? Which nearby mandi (market) is offering the best price for my cotton crop?
While agricultural data exists across various portals, accessing it through complex web interfaces or text-heavy apps is challenging for farmers out in the field.
Voice is the natural, frictionless interface for Bharat. A farmer standing in an orchard or driving a tractor doesn't want to type queries into a search bar; they want to speak naturally in their native language or conversational Hinglish and get instant, reliable answers.
For the 10 Days of Voice Agents (VoiceForBharat Edition), I chose the Farm & Field track and built Kisan Mitra (किसान मित्र) — an empathetic, real-time AI voice assistant tailored specifically for Indian agriculture.
🏗️ Architecture & Core Components
A production-grade voice agent is fundamentally different from a text chatbot. Latency is the single biggest factor in conversational realism: if the agent takes more than 1–1.5 seconds to reply, the human conversation breaks down.
mermaid
flowchart LR
A[🎙️ Farmer Speaks] -->|Audio Stream| B(Deepgram Nova-3 STT)
B -->|Transcribed Text| C(Gemini 2.5 Flash LLM)
C -->|Streamed Tokens| D(Murf Falcon TTS)
D -->|Real-time Audio| E(LiveKit WebRTC)
E -->|Ultra-low Latency Audio| F[🔊 Farmer Hears Answer]
C <-->|Tools & Memory| G[(SQLite & External APIs)]
The 4 Pillars of the Pipeline:
Real-time Transport (LiveKit): Manages ultra-low-latency, bidirectional audio WebRTC streaming and turn detection.
Speech-to-Text (Deepgram Nova-3): Accurately transcribes spoken Indian English and accented Hindi.
LLM Brain (Google Gemini 2.5 Flash): Handles intent detection, domain reasoning, guardrails, and tool calling.
Fast Text-to-Speech (Murf Falcon): The game changer. With sub-100ms time-to-first-audio, Murf Falconstreams natural, warm Indian voices (Anisha / hi-IN) without robotic pauses.
🚀 Key Features Built Across the 10 Days
- Authentic Indian Voice & Conversational Guardrails Using Murf Falcon's hi-IN voice with conversational styling and sentence tokenization, Kisan Mitra sounds warm, respectful, and native. The system prompt strictly enforces:
Conversational, concise spoken responses (no markdown syntax or raw JSON read aloud).
Absolute refusal to fabricate market rates or weather data.
- Live Agricultural Tools (Mandi Prices & Open-Meteo Weather) Kisan Mitra is armed with domain tools that fetch real-world data:
lookup_mandi_prices(crop, district): Queries market prices across key Indian APMCs (e.g., Yavatmal, Nagpur, Lasalgaon) and explicitly cites timestamps (e.g., "As of today's Agmarknet live update...").
get_district_weather(district): Leverages Open-Meteo live satellite feeds to deliver temperature, rain probability, and actionable agronomic advice (e.g., "Rain probability is 65% today; postpone chemical spraying").
Graceful Failure & Out-Loud Transparency
External APIs fail in the real world. Rather than hallucinating rates or hanging silently, Kisan Mitra catches timeouts and announces the service outage out loud to the caller:Caller Memory & Privacy-First Persistence
Kisan Mitra remembers returning farmers (e.g., their land size, crops grown, district) across sessions via SQLite. Crucial guardrail: The agent never saves data without first asking: "May I save these details so I can remember you for our next call?"Outbound Telephony & Proactive Price Alerts
When mandi rates cross a farmer's predefined threshold (e.g., Cotton crossing ₹7,000/quintal in Yavatmal), Kisan Mitra autonomously places a phone call via Twilio & TwiML:
Compliance in the first 2 sentences: Explains who is calling, why, and how to opt out (Press 9 to unsubscribe, 1 for details).
Outcome tracking: Automatically handles busy lines, no-answers (retry in 2h), and short hang-ups.
Human Escalation (Krishi Vigyan Kendra Officer Support)
For emergencies (severe pest attacks like Pink Bollworm or crop blight), the agent prompts the farmer for consent and logs a structured escalation ticket with sanitized PII, assigning a reference number like ESC-48291 for agricultural officer callbacks.Specialist Agent Handoffs & Analytics Dashboard
Handoffs: Complex agronomic pathology queries are handed off seamlessly to a dedicated CropSpecialist sub-agent.
Analytics: Complete call outcome tracking (completed, success, failed, reason) stored in SQLite and visualizable via an admin dashboard.
🛠️ The Hardest Challenges & Lessons Learned
Eliminating Conversational Latency
The Problem: Combining STT + LLM reasoning + TTS synthesis often introduces awkward 2–3 second silences.
The Solution: Streaming tokens incrementally from Gemini into Murf Falcon using SentenceTokenizer(min_sentence_len=2) and preemptive_generation=True. Audio synthesis begins before the LLM finishes generating the full paragraph.Handling Code-Mixed Hindi & Indian English (Hinglish)
The Problem: Standard VAD (Voice Activity Detection) models often cut off speakers mid-sentence when Indian language filler words ("haanji", "achha", "matlab") were used.
The Solution: Integrated LiveKit's MultilingualModel turn detector paired with Silero VAD to maintain natural listening rhythms.Out-Loud Tool Failures vs. Silent Errors
The Problem: When an external weather API timed out, the LLM initially tended to guess yesterday's temperature.
The Solution: Enforced strict system instructions where tools return explicit FAILURE strings that instruct the LLM: "State out loud that the service is unreachable right now. Do not guess."
GITHUB :- https://github.com/codebynikhil08/murf-livekit-starter
