Building Vidya: A Production-Grade Multi-Agent Voice Cyber-Tutor with Murf Falcon & LiveKit

A complete engineering retrospective on building an ultra-low-latency voice AI tutor with specialist handoffs, vector RAG, SQLite memory, live tools, telephony, and real-time analytics for the Murf 10 Days Voice Agent Challenge (#VoiceForBharat).

1. The Problem: Voice-First Learning for Bharat

In many regions across Bharat (India), millions of students and adult learners face a dual barrier: digital literacy hurdles and fear of judgment when asking basic questions. Traditional screen-based interfaces require typing, navigating complex menus, and reading dense text — creating unnecessary friction. Moreover, classroom environments often intimidate students who struggle with foundational concepts like arithmetic, grammar, or digital safety (UPI payments, OTP scams, phishing).

Voice changes everything.

Conversational voice AI provides a zero-friction, patient, and infinitely encouraging tutor that meets students where they are. That was my mission for the Learning & Literacy track of the Murf 10 Days Voice Agent Challenge: to build Vidya, an intelligent Cyber-Tutor that speaks with a natural Indian accent, remembers learners across sessions, pulls verified facts from curriculum vector databases and live APIs, hands off complex math to a dedicated specialist, and even places proactive telephony practice calls.

2. System Architecture: How It All Works

Building a real-time conversational agent requires orchestrating four critical layers with sub-second latency:

flowchart TD
    subgraph ClientLayer["🖥️ Frontend Client (Next.js 16 + React 19)"]
        UI["Cyberpunk Matrix Grid Audio Visualizer (30x30)"]
        Dash["📊 Real-Time Call Telemetry Dashboard (/dashboard)"]
    end

    subgraph TransportLayer["⚡ Real-Time Transport (LiveKit WebRTC)"]
        LK["LiveKit Audio & Data Pipeline"]
    end

    subgraph AgentLayer["🧠 Multi-Agent Voice Pipeline"]
        VAD["Silero VAD + Turn Detector"]
        STT["Deepgram Nova-3 (en-IN)"]

        subgraph Agents["Specialist Handoff Engine"]
            Vidya["Primary Agent: Vidya\n(General & Literacy Tutor)\nMurf Voice: Anisha"]
            Kash["Specialist Agent: Kash\n(Advanced Mathematics)\nMurf Voice: en-US-matthew"]
            Vidya -.->|Dynamic Silent Handoff\nwith ChatContext| Kash
        end

        TTS["Murf Falcon Streaming TTS\n(55ms Latency)"]
    end

    subgraph Subsystems["💾 Knowledge, Memory & Tooling"]
        RAG["ChromaDB Vector Store\n(Curriculum RAG)"]
        MemDB["SQLite agent_memory.db\n(Student Profile & Analytics)"]
        Tools["Live Tools:\n• Wikipedia API\n• Dictionary API\n• ExchangeRate API\n• Discord Escalation\n• SIP Softphone"]
    end

    UI <-->|WebRTC Stream| LK
    LK <--> VAD
    VAD --> STT
    STT --> Vidya
    Vidya <--> RAG
    Vidya <--> MemDB
    Vidya <--> Tools
    Vidya --> TTS
    Kash --> TTS
    TTS --> LK
    MemDB -.->|Auto-sync| Dash

The Pipeline in Numbers

• Streaming TTS: Murf Falcon with 55ms model latency and 130ms Time-to-First-Audio (TTFA).
• STT: Deepgram Nova-3 optimized for en-IN (Indian English).
• LLM: Local Ollama (gemma4:31b-cloud), Google Gemini (gemini-2.5-flash), or OpenAI (gpt-4o).
• Transport: LiveKit WebRTC audio transport with Silero VAD and Multilingual Turn Detection.

3. Key Features Built Across the 10-Day Journey

1. Ultra-Low Latency Indian Voice with Murf Falcon

Using Murf Falcon's streaming TTS, Vidya speaks with Anisha — a warm, natural Indian English voice. To eliminate conversational pauses:

• Tokenizer sentence threshold tuned to min_sentence_len=10.
• text_pacing=False to strip synthetic delays.
• Preemptive generation enabled on LiveKit sessions.

2. Multi-Agent Specialist Handoff (Vidya $\rightarrow$ Kash)

When a student asks for higher-level mathematics (calculus, trigonometry, algebra drills), Vidya dynamically executes a specialist handoff to Kash (voiced by Murf Falcon's en-US-matthew):

@function_tool(
    description="Transfer the user to the Mathematics Specialist. Call this ONLY when the user asks for advanced math practice or has a complex mathematical question."
)
async def transfer_to_maths_specialist(self, ctx: RunContext) -> Agent:
    logger.info("Transferring to Maths Specialist")
    # Propagate the full conversation context so the user doesn't repeat themselves!
    return MathsSpecialist(
        room=self.room,
        chat_ctx=ctx.session._chat_ctx
    )

3. Curriculum RAG with ChromaDB

To prevent hallucinations on syllabus topics (digital safety laws, grammar rules, arithmetic), curriculum markdown documents in backend/data/ are chunked and ingested into a local ChromaDB vector collection (syllabus). When queried, Vidya grounds her explanations in verified curriculum context.

4. Silent Long-Term Memory (SQLite)

Vidya tracks user facts, names, and interaction histories in agent_memory.db. When you reconnect, Vidya retrieves your profile before speaking:

"Hello Farhan, how are you? Welcome back to our English practice!"

5. Live Tool Integrations & Chaos Resilience

• Wikipedia Encyclopedia API: Instant retrieval of historical and scientific facts.
• Free Dictionary API: Word definitions, parts of speech, and pronunciation examples.
• Real-Time FX Rates (api.exchangerate-api.com): Live currency math problems with date stamps and graceful offline fallback if network calls fail.

6. Human Escalation Support (Discord Webhook)

If a student gets severely frustrated or explicitly asks for a human teacher, Vidya confirms consent, generates a support reference ID (VIDYA-TKT-XXX), and pushes an alert directly to a Discord teacher channel.

7. Telephony (Inbound & Outbound SIP)

• Inbound Agent: Answers phone calls via LiveKit SIP trunks, extracts caller ID, applies BVCTelephony noise cancellation, and allows warm human transfers.
• Outbound Agent: Dials proactive practice calls, detects answering machines, and handles SIP error codes (486 Busy $\rightarrow$ 1h retry, 408 No Answer $\rightarrow$ 4h retry, 603 Declined $\rightarrow$ cancel & log to SQLite).

8. Cyberpunk Telemetry Dashboard (/dashboard)

A dedicated Next.js dashboard polls /analytics.json (auto-synced from SQLite) every 5 seconds to track total sessions, successful learning uplinks, and dropped calls in real time.

4. Engineering Challenges & How I Solved Them

Challenge 1: The Multi-Language STT Latency Trap

• Problem: Initially, using multi-language STT (language="multi") caused noticeable speech-recognition lag, creating awkward conversational gaps.
• Root Cause: Universal multilingual models are significantly heavier and require extra time to classify the language before transcribing.
• Solution: Explicitly locked Deepgram STT to language="en-IN" with smart_format=True. STT latency dropped by over 40%, making turn-taking instantaneous.

Challenge 2: Context Wipe During Specialist Agent Handoff

• Problem: When Vidya handed off to Kash, the math specialist started with a blank slate, asking the student to repeat what problem they were working on.
• Solution: Passed chat_ctx=ctx.session._chat_ctx from the parent agent's RunContext directly into the MathsSpecialist constructor. Kash seamlessly picked up the exact sentence and numbers the student was discussing.

Challenge 3: Live API Hangs in Real-Time Voice Loops

• Problem: External HTTP calls (like exchange rate APIs) can hang on flaky connections. A 10-second network delay in a voice agent feels like an eternity.
• Solution: Wrapped external requests in strict 5-second socket timeouts with custom try/except error recovery blocks. Created chaos test scripts (break_api.py and fix_api.py) to verify that Vidya gracefully apologizes and steers the student to another topic instead of hanging.

5. Build Your Own Voice Agent: Step-by-Step Guide

Want to build your own real-time voice agent? Here is the exact blueprint.

1. Prerequisites

• Python 3.10+ & uv
• Node.js 18+ & pnpm
• Accounts & API keys for:
  • Murf Falcon (Streaming TTS)
  • Deepgram (STT)
  • LiveKit Cloud (WebRTC Transport)
  • Google AI Studio or OpenAI (LLM)

2. Clone the Repository

git clone https://github.com/your-org/vidya-ai-tutor.git
cd vidya-ai-tutor

3. Configure Environment Variables

Create .env.local in backend/ and frontend/:

# LiveKit Credentials
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

# Voice & AI Keys
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key

# Optional Integrations
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...

4. Ingest Knowledge & Download Models

cd backend
uv sync
uv run python src/rag_ingest.py              # Ingest curriculum into ChromaDB
uv run python src/agent.py download-files    # Download Silero VAD models

5. Launch All Services

• Windows (with High-Performance CPU tuning):

  .\start_app.ps1

• macOS / Linux:

  chmod +x start_app.sh
  ./start_app.sh

Open http://localhost:3000 in your browser, click INITIATE LEARNING MODULE, and speak into your microphone!

6. What's Next for Vidya?

• Real-Time Video Screen Annotation: Adding LiveKit video input so Vidya can visually inspect handwritten math equations on a student's notebook.
• Multilingual Vernacular Voices: Expanding into Hindi, Tamil, Telugu, and Bengali voice models using Murf Falcon's multilingual library.
• Automated WhatsApp Progress Reports: Dispatching summary cards and practice reminders to parents after each voice tutoring session.

7. Links & Repository

• GitHub Repository: https://github.com/your-org/vidya-ai-tutor
• Murf Falcon TTS Docs: https://murf.ai/api/docs/text-to-speech/streaming
• LiveKit Agents SDK: https://docs.livekit.io/agents

Built with passion during the #VoiceForBharat 10 Days Voice Agent Challenge powered by Murf AI.