Building an AI Interview Coach with VoiceForBharat: A 10-Day AI voice agent journey

How I built a multi-agent, multilingual AI Interview Coach powered by Murf Falcon, LiveKit, Deepgram, and Gemini — complete with caller memory, real-time guardrails, outbound calling, human escalation, and call analytics.

1. Introduction & Problem Statement

Job interview preparation in India is often high-stakes, intimidating, and inequitable. While mock interviews with senior engineers or professional coaches cost thousands of rupees per session (typically ₹1,000–₹3,000 per hour), millions of job seekers in India—especially engineering students and candidates from Tier-2/Tier-3 cities—struggle with technical confidence, articulation, and language barriers when balancing English and Hinglish.

To solve this, I participated in 10 Days of Voice Agents — VoiceForBharat Edition organized by Murf AI. Over nine intensive days, I built Anisha (and her technical companion Dev), an AI Interview Coach under the Learning & Literacy track.

Target Audience & Core Pain Points

• Target Audience: Engineering graduates, job seekers, Tier-2/Tier-3 college students, and career upskillers in India preparing for technical, behavioral, and DSA (Data Structures & Algorithms) interview rounds.
• The Problem It Solves:
  1. High Cost & Lack of Mentorship: Professional mock interview platforms are financially out of reach for students needing daily practice.
  2. The Articulation & Anxiety Gap: Candidates frequently possess strong theoretical coding knowledge but freeze up or struggle when explaining their problem-solving steps out loud under real-time pressure.
  3. Rigid Language Expectations (English vs. Hinglish): Candidates often feel most comfortable explaining complex logic in Hinglish (code-mixed Hindi + English), whereas conventional platforms enforce strict English-only environments.

Why Voice over Text Chatbots?

Interviews are fundamentally oral, dynamic, high-latency human interactions! Text-based LLM chatbots (like ChatGPT) cannot simulate real-world vocal pressure, tone, speech pacing, or natural spoken interruptions. A real-time voice agent provides immediate conversational feedback, allowing candidates to practice spoken flow, eliminate filler words, and build true interview-day confidence.

2. How the System Works: High-Level Architecture & Pipeline

Building an interactive voice agent requires orchestrating four key building blocks in near-zero latency:

┌─────────────────┐       ┌─────────────────┐       ┌──────────────────┐       ┌──────────────────┐
│  🎙️ User Speech  │ ────> │  Deepgram STT   │ ────> │ Gemini 2.5 Flash │ ────> │ Murf Falcon TTS  │ ────> 🔊 User Hears
│  (Microphone)   │ Audio │ (Nova-3 / Auto) │ Text  │ (LLM & Logic)    │ Text  │ (55ms Latency)   │ Audio
└─────────────────┘       └─────────────────┘       └──────────────────┘       └──────────────────┘
                                                             │
                                                  LiveKit Real-Time WebRTC
                                            ┌────────────────────────────────┐
                                            │ SQLite Memory / DB & Dashboard │
                                            └────────────────────────────────┘

The Real-Time Audio Loop

1. Speech-to-Text (STT): Deepgram Nova-3 transcribes live audio in real-time with multi-lingual auto-detection support (English & Hinglish), capturing candidate speech with high accuracy.
2. Brain & Logic (LLM): Gemini 2.5 Flash acts as the agent's brain, processing conversation context, evaluating candidate answers, enforcing safety guardrails, and triggering function tools.
3. Text-to-Speech (TTS): Murf Falcon streams ultra-fast (55ms latency), natural Indian English audio (Anisha voice) back to the user.
4. Real-Time Transport & Orchestration: LiveKit manages WebRTC audio streaming, frame synchronization, VAD (Voice Activity Detection), and turn management between the browser/phone and Python backend.
5. Persistence Layer: SQLite (caller_memory.db) persists user profiles, past interview history, human escalation tickets, and call logs.

3. Deep Dive into Key Features Built Across the 10 Days

Here is a comprehensive breakdown of the core capabilities powering the AI Interview Coach:

1. Indian Voice & Cultural Persona (Anisha)

Powered by Murf Falcon's Anisha voice, the agent delivers warm, professional, and encouraging Indian English pronunciation. The system defaults to clear English but dynamically adapts to code-mixed Hinglish when candidates express themselves in Hindi/Hinglish, rendering Hinglish text in clean Roman script for pristine Murf TTS playback.

2. Safety Guardrails & Consent Protocols

• Data Protection: Strict system guardrails prohibit collecting sensitive personal data like Passwords, Aadhaar, PAN numbers, or banking credentials.
• Do-Not-Call / Opt-Out Support: Candidates can state "Stop calling me" or "Opt out". The agent immediately calls opt_out_caller(user_id), updates the database, and terminates phone outreach.

3. Long-Term Caller Memory & Database Persistence (db.py)

Returning candidates are greeted by name ("Hello Ramesh, welcome back! Last time we practiced your Python interview..."). The agent uses get_caller() and upsert_caller() to track:

• Candidate Name & Language Preference (Hinglish/English)
• Target Role (e.g., Frontend Engineer, Backend Python Developer)
• Skill level (Fresher, Mid-Level) & weak areas noted during prior practice sessions

4. Live API Tool Integration (lookup_interview_question)

When candidates request a coding or behavioral prompt, the agent invokes lookup_interview_question(). This tool queries the public Tech Interview Handbook repository on GitHub API to retrieve live, up-to-date exercises.

• Freshness Transparency: Explains data freshness out loud (e.g., "From Tech Interview Handbook updated as of 29 July 2026").
• Graceful Failover: If the network request times out, it switches instantly to a verified local cache (status: partial_offline_fallback) without awkward silences.

5. Outbound Phone Calling & SIP Telephony (outbound_call.py)

Candidates can schedule daily practice calls. Using LiveKit SIP and Twilio/Linphone integration, the script dispatches outbound calls directly to the candidate's phone:

• Mandatory Opening Protocol: The agent announces who it is, why it's calling, and explicitly explains how to opt out in the very first two sentences.

6. Human Escalation Ticket Workflow (create_escalation)

When candidates express severe interview distress or explicitly request senior human mentor assistance (e.g., placement guarantees or human resume review):

• The agent asks for explicit consent before logging details.
• It invokes create_escalation() to write a ticket to caller_memory.db.
• It speaks a tracking Reference ID (e.g., ESC-8A3F21) to the caller and promises a human follow-up within 24 hours.

7. Multi-Agent Specialist Handoff (DSASpecialist / Dev)

When an interview transitions from general behavioral prep to Data Structures & Algorithms:

• Anisha asks: "I can connect you to our DSA practice expert, Dev, who specializes in technical coding problems. Would you like me to connect you to Dev now?"
• Upon candidate agreement, the agent triggers handoff_to_dsa_specialist().
• State transfers seamlessly to Dev (DSASpecialist), who greets the candidate ("Hello! I am Dev, your DSA practice expert...") and continues with full candidate context without asking the user to repeat themselves.

8. Privacy-Compliant Call Analytics Dashboard (/dashboard)

Built into the Next.js frontend, the dashboard tracks real-time session performance for the Learning & Literacy track:

• Metrics Tracked: Total Calls, Successful Completed Interviews, Failed/Dropped Calls, and Overall Success Rate %.
• Privacy Shield: Strictly aggregates counts while automatically filtering out caller transcripts, passwords, phone numbers, and private data.

4. Difficult Challenges Faced & How They Were Solved

Challenge 1: Audio Overlap & Interruption Latency

• The Problem: In early builds, when a candidate interrupted the agent mid-sentence, the agent's TTS stream continued playing for 1–2 seconds, causing overlapping audio and chaotic conversations.
• The Root Cause: High latency in silence detection and turn-taking signals created a mismatch between streaming audio frames and user voice activity detection.
• The Solution: Combined LiveKit's MultilingualModel turn detector with Murf Falcon's ultra-low 55ms streaming TTS latency. Because Murf Falcon delivers time-to-first-audio (TTFA) in milliseconds, the pipeline can halt audio playback instantly the moment user speech is detected.

Challenge 2: Network Delays During Real-Time Tool Calling

• The Problem: Fetching live interview questions from external APIs mid-session caused 3+ second pauses, leading users to ask "Are you still there?".
• The Solution: Implemented a strict 4-second timeout on network socket calls in Python and created a fallback mechanism (status: partial_offline_fallback) that returns a pre-cached offline question set, allowing the agent to answer immediately without breaking conversation flow.

5. Step-by-Step Guide: How to Build and Run Your Own Agent

Follow these practical steps to spin up your own Murf Falcon + LiveKit voice agent!

Step 1: System Prerequisites

• Python 3.10+ with uv package manager installed
• Node.js 18+ with pnpm
• LiveKit Server account (LiveKit Cloud or local CLI)

Step 2: Environment Configuration

Create a .env.local file in both backend/ and frontend/ directories:

# LiveKit WebRTC Config
LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

# AI Service Keys
MURF_API_KEY=your_murf_falcon_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_gemini_api_key

Security Notice: Never commit your .env.local files or API keys to public repositories. Ensure .env.local is listed in your .gitignore.

Step 3: Backend Agent Setup & Execution

cd backend
# Install dependencies with uv
uv sync

# Pre-download required LiveKit model files
uv run python src/agent.py download-files

# Start agent in development mode
uv run python src/agent.py dev

Step 4: Frontend UI Setup & Execution

cd frontend
# Install Node dependencies
pnpm install

# Start Next.js dev server
pnpm dev

Step 5: Test the Voice Agent

1. Open http://localhost:3000 in your browser.
2. Click "Start talking" and grant microphone permissions.
3. Greet Anisha ("Hi Anisha, I want to practice for a Python developer interview").
4. Ask to practice DSA questions to test the handoff to Dev!

🔮 6. Future Enhancements

1. Multimodal Code Canvas: Integrating a side-by-side Monaco code editor into the Next.js UI, allowing Dev to evaluate written code syntax while giving verbal feedback.
2. Regional Language Expansion: Adding native voice support for regional Indian languages (such as Hindi, Tamil, Telugu, and Kannada) available in Murf Falcon's voice library.

🔗 7. Code Repository & Links

• Public GitHub Repository: https://github.com/Ayontikapal/murf-livekit-starter/tree/day9
• Murf Falcon TTS API Docs: https://murf.ai/api/docs/text-to-speech/streaming
• Murf Voice Library: https://murf.ai/api/docs/voices-styles/voice-library