Jan Sahay: Building a Voice AI Agent for Government Schemes & Financial Guidance

The Problem

Understanding government schemes and basic financial services can be difficult for many people. Users may have questions about eligibility, benefits, required documents, bank accounts, savings, or how to access the right information.

Jan Sahay is my voice-based AI assistant designed to make this information easier to access through a natural conversation. Instead of searching through multiple websites or forms, users can simply speak to Jan Sahay and ask their questions.

The agent is designed for citizens who need simple and accessible guidance about government schemes and basic financial services. Voice is useful here because users can interact naturally without having to type long questions or navigate complicated interfaces.

This project is part of 10 Days of Voice Agents — VoiceForBharat Edition.

What Jan Sahay Does

Jan Sahay is built as a voice-first financial assistance agent. It can help users with government schemes as well as basic banking and financial guidance.

Some of the important capabilities I built include:

• Natural voice conversations
• Government scheme information and guidance
• Basic banking and savings guidance
• Safety guardrails for sensitive financial information
• Memory for returning users
• Tools and database integration
• Outbound voice calls
• Human escalation when needed
• Call analytics and history
• Specialist-agent handoff for government scheme queries

The idea is to keep the conversation simple for the user while allowing the system behind the scenes to handle more complex tasks.

How the Voice Agent Works

The voice interaction follows a real-time pipeline:
User speaks
→ Speech-to-Text
→ Main AI Agent
→ Tools, memory, or specialist handoff when required
→ Text-to-Speech
→ User hears the response

The main components of the system are:

• Speech-to-Text: Deepgram converts the user's speech into text.
• LLM / AI Agent: The Jan Sahay agent understands the request, follows its instructions and safety guardrails, and decides how to respond.
• Text-to-Speech: Murf Falcon converts the agent's response into natural-sounding speech.
• Real-time Transport: LiveKit handles the real-time voice communication between the user and the agent.

The frontend is built with Next.js, while the backend uses Python. SQLite is used for storing relevant application and call data.

Key Features I Built

Safety Guardrails

Jan Sahay follows safety-focused instructions for financial conversations. It does not ask users to share sensitive information such as OTPs, PINs, passwords, UPI PINs, CVVs, or full card and bank account numbers.

Memory

The agent can remember relevant user information so that returning users can have more useful conversations without having to repeat everything.

Tools and Database

The backend uses tools and SQLite to store and retrieve relevant application and call information. This also supports the call analytics dashboard.

Outbound Calls

Jan Sahay includes outbound calling functionality, allowing the system to initiate a voice call when required.

Human Escalation

When a request needs human assistance, the system can escalate the conversation instead of trying to handle everything through the AI.

Call Analytics Dashboard

I built a dashboard to track real voice-call activity, including:

• Total calls
• Successful calls
• Failed calls
• Success rate
• Average call duration
• Recent call history

The dashboard uses data from actual calls rather than hardcoded values.

Specialist Agent Handoff

The main Jan Sahay agent does not try to handle every type of request.
When a user asks specifically about an Indian government financial scheme, Jan Sahay tells the user that it will connect them to the Government Scheme Specialist.
The specialist then continues the conversation using the existing context.

For example:
User: I need help with a government scheme.
Jan Sahay: I'll connect you to our Government Scheme Specialist.
Specialist: Hello! I'm your Government Scheme Specialist. How can I help you today?

The user can then continue asking questions such as PMJDY eligibility, benefits, or required documents without repeating the original request.

Challenges I Faced

Building Jan Sahay was not always smooth. Throughout the challenge, I faced several technical and conversation-flow problems. Solving these issues helped me understand how different parts of a voice AI system work together.

1. Initial Setup and Voice Connection

During Day 1, getting the initial voice agent setup working was challenging. Sometimes the agent did not join the LiveKit session, or I could not start a proper conversation with the agent.
I had to check the backend process, session connection, environment configuration, and voice pipeline to identify the problem. Re-running the agent and checking the logs helped me understand where the connection was failing.

2. Memory

Implementing memory was another challenge. The agent needed to remember relevant information about returning users without making the conversation feel unnatural.
I had to work on how information was stored and retrieved so that the agent could use previous context appropriately.

3. Outbound Calls

Getting outbound calling to work reliably required additional testing. The call connection, configuration, and agent availability all needed to work together.
Testing different call scenarios helped me understand that an outbound voice feature needs more than just triggering a phone call—the agent also needs to be ready to handle the conversation once the call connects.

4. Human Escalation

Human escalation was another important challenge. The agent needed to recognize when a situation should be handled by a human instead of continuing to answer on its own.
I had to test the escalation flow and make sure the user could be moved toward human assistance without making the conversation confusing.

5. Call Analytics Dashboard

The dashboard also required several rounds of testing.
I needed to make sure that successful and failed calls were recorded correctly and that metrics such as total calls, success rate, average call duration, and recent call history were updated from actual call data rather than hardcoded values.
Testing both successful and failed call paths helped me verify that the dashboard reflected real call activity.

6. Specialist Agent Handoff

The specialist-agent handoff was one of the more challenging parts of the project.
Initially, I faced a circular import issue between the main agent and the specialist agent:

Main Agent → Specialist Agent → Main Agent

This caused an import error when starting the backend.
I solved it by separating the specialist agent properly and removing the unnecessary dependency on the main agent.
I also had to improve the conversation flow because the handoff message could sometimes be repeated. I adjusted the instructions so that Jan Sahay clearly announces the handoff and the Government Scheme Specialist takes over naturally.

What These Challenges Taught Me

These problems taught me that building a voice AI agent is not only about connecting an LLM to speech.
The real-time session, speech pipeline, memory, tools, database, outbound calling, human escalation, analytics, and multi-agent architecture all have to work together.
Most importantly, I learned that testing real conversations is essential. A system that looks correct in code can still behave differently during an actual voice interaction.

How to Build and Run Your Own Voice Agent

If you want to build a similar voice agent, the basic architecture can be broken into four main parts:

1. Speech-to-Text

The user's voice needs to be converted into text so that the AI agent can understand it.

In my project, I used Deepgram for speech-to-text.

2. LLM / AI Agent

The transcribed text is passed to the AI agent. The agent understands the user's request, follows its instructions and safety guardrails, and decides whether to answer directly, use a tool, or hand the conversation to a specialist.

3. Text-to-Speech

The agent's text response is converted back into natural speech.

I used Murf Falcon for the text-to-speech layer.

4. Real-Time Voice Transport

A real-time communication layer is needed to connect the user's voice with the agent.

I used LiveKit to handle the real-time voice session.

The overall flow looks like this:

User Voice
→ Deepgram Speech-to-Text
→ AI Agent / LLM
→ Tools, Memory or Specialist Handoff
→ Murf Falcon Text-to-Speech
→ User hears the response

Setting Up the Project

First, clone the public repository:

git clone https://github.com/lipsapay7-collab/jan-sahay-voice-ai.git
cd jan-sahay-voice-ai

Running the Project

Start the local LiveKit server in Terminal 1:

livekit-server --dev

In Terminal 2, start the Jan Sahay backend:

uv run python src/agent.py dev

In Terminal 3, start the Next.js frontend:

pnpm dev

Once all three are running, open the local Jan Sahay application in the browser and start a voice call.

The basic local setup is:

LiveKit Server
→ Jan Sahay Backend
→ Next.js Frontend
→ Voice Conversation

Keeping API Keys Safe

Jan Sahay uses environment variables to store API keys and service credentials.

My backend environment includes credentials for:

• LiveKit for real-time voice transport
• Murf for text-to-speech
• Deepgram for speech-to-text
• Google Gemini for the LLM
• SIP configuration for outbound calling

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key

LINPHONE_SIP_URI=your_sip_uri
SIP_OUTBOUND_HOST=your_sip_host
LIVEKIT_SIP_TRUNK_ID=your_sip_trunk_id
OTEL_SDK_DISABLED=true

These values should be stored in a local environment file such as .env.local or another environment file supported by the project.

Testing the Conversation

After starting the LiveKit server, backend, and frontend, I tested the agent through real voice conversations.

For a normal question:
User: What is a savings account?
The request stays with the main Jan Sahay agent.

For a government scheme question:
User: I need help with a government scheme.
Jan Sahay informs the user that the conversation will be handed to the Government Scheme Specialist.

The specialist then continues the conversation using the existing context.

I also tested memory, outbound calls, human escalation, call analytics, successful and failed call paths, and specialist handoffs during development.

Evidence From My Build

Throughout the 10 Days of Voice Agents challenge, I documented the major stages of building Jan Sahay.

Day 1 — Getting the Voice Agent Working

I started by setting up the basic real-time voice agent and connecting the required voice services.

During the initial setup, I faced issues such as the agent sometimes not joining the LiveKit session and the voice conversation not always starting correctly.

Day 1 Evidence:

Day 2 — Personality, Job and Safety Guardrails

I gave Jan Sahay a clear identity, purpose, responsibilities, and safety boundaries.

The agent was instructed to provide financial guidance safely and avoid asking users for sensitive information such as OTPs, PINs, passwords, CVVs, or other confidential credentials.

Day 2 Evidence:

Day 3 — Frontend Personalisation

I created and personalised the Jan Sahay frontend so users can start a voice conversation through a simple interface.

The frontend also shows the main capabilities of the application, including government schemes, fraud prevention, financial literacy, and complaint helplines.

Day 3 Evidence:

Day 4 — Memory

I added memory so that the agent can use relevant information from previous interactions instead of treating every conversation as completely new.

This helped make conversations with returning users more personalised.

Day 4 Evidence:

Day 5 — Tools

I added tools to make the agent more useful than a basic conversational AI.

For example, the system can support financial-service tasks such as scheme eligibility checks based on collected answers and document-checklist guidance.

Day 5 Evidence:

Day 6 — Outbound Calls

I implemented outbound calling so that Jan Sahay can initiate a voice call when required.

This involved connecting the voice agent with the calling infrastructure and testing whether the agent could handle the conversation after the call connected.

Day 6 Evidence:

Day 7 — Human Help and Escalation

I added a human-escalation flow for situations where the AI should not try to handle everything itself.

The goal was to make sure the user could be guided toward human assistance when necessary.

Day 7 Evidence:

Day 8 — Call Analytics Dashboard

I built a call analytics dashboard using real call data.

It tracks:

• Total Calls
• Successful Calls
• Failed Calls
• Success Rate
• Average Call Duration
• Recent Call History

I tested both successful and failed call paths and verified that the dashboard updates from actual call activity rather than hardcoded values.

Day 8 Evidence:

Day 9 — Specialist Agent Handoff

On Day 9, I added a dedicated Government Scheme Specialist Agent.

The main Jan Sahay agent handles general questions, but when the user asks specifically about an Indian government financial scheme, it hands the conversation to the specialist.

The user is informed before the handoff, and the specialist continues the conversation using the existing context.

The user can then continue asking about eligibility, benefits, documents, or application information without repeating the original question.

Day 9 Evidence:

My Development Journey

Each day added another layer to Jan Sahay:

Voice Agent → Personality & Guardrails → Frontend → Memory → Tools → Outbound Calls → Human Escalation → Analytics → Specialist Handoff

Together, these features turned Jan Sahay from a basic voice assistant into a more complete voice AI system.

What I Would Improve Next

Although Jan Sahay is working as a functional voice AI prototype, there are several things I would improve in the future.

• Add support for more Indian languages and regional language conversations.
• Connect the agent to more reliable and up-to-date government scheme data sources.
• Improve voice-session reliability and automatic recovery when the agent fails to join a session.
• Make specialist handoffs smoother and more natural.
• Improve memory so that only useful and relevant information is retained.
• Add more detailed call analytics and monitoring.
• Improve the frontend accessibility and user experience.
• Add stronger automated testing for different voice-call scenarios.
• Prepare the system for more reliable production deployment.

What I Learned

This 10-day challenge gave me practical experience in building a complete voice AI application.

I learned that a voice agent is not just an LLM with text-to-speech. A reliable voice application requires real-time communication, speech-to-text, text-to-speech, agent instructions, memory, tools, databases, calling infrastructure, safety guardrails, analytics, and clear agent responsibilities.

I also learned the importance of testing with real conversations. Some issues were difficult to identify from the code alone, such as the agent not joining a LiveKit session, conversations not continuing properly, and handoff messages being repeated.

The biggest lesson I learned was to build the system step by step and test each feature before adding the next one.

Building Jan Sahay throughout these 10 days helped me understand both the technical and practical challenges of creating voice AI for real users.

Project Links

GitHub Repository: https://github.com/lipsapayal7-collab/jan-sahay-voice-ai
LinkedIn: www.linkedin.com/in/lipsarani-sahoo-235082379