Yes โ now I understand exactly what you mean.
The written content you provided is your friend's blog. The screenshots are from your actual Jan Sahay implementation. So I will use your friend's article only as a reference for structure/storytelling, but the blog below will be written around your actual UI and features shown in the screenshots.
I also won't claim things like "sub-500ms" or specific backend implementation details unless they are actually part of your project.
Jan Sahay: Building a Voice AI Assistant for Bharat ๐ฎ๐ณ
10 Days of Voice Agents โ VoiceForBharat Edition
What started as a simple voice agent on Day 1 slowly became something much bigger.
For the 10 Days of Voice Agents โ VoiceForBharat Edition, I decided to build Jan Sahay (เคเคจ เคธเคนเคพเคฏเค), a voice-first AI assistant focused on financial services, government schemes, digital fraud awareness, and citizen support.
The idea was simple:
Make important financial and government information easier to access through a natural voice conversation.
Instead of expecting users to search through multiple websites or understand complicated financial terminology, Jan Sahay lets them simply talk to an AI assistant.
Over the ten days, I gradually added more capabilities โ from basic voice conversations to memory, real-world tools, outbound calling, human escalation, call analytics, and specialist agents.
This blog is a summary of that journey.
๐๏ธ What is Jan Sahay?
Jan Sahay is designed as a citizen-focused AI voice assistant for the Financial Services track.
It can help users explore areas such as:
- ๐๏ธ Government schemes
- ๐ก๏ธ Digital fraud prevention
- ๐ฐ Financial literacy
- ๐ Complaint and escalation guidance
- ๐๏ธ Voice-based conversations
- ๐ค Specialist-agent assistance
The main goal isn't to replace government officials, banks, or financial experts.
It is to provide a simple first point of assistance โ helping users understand their options and guiding them toward the appropriate next step.
๐๏ธ Why Voice?
One of the biggest reasons I chose voice is accessibility.
Not everyone wants to type a long question into a search box.
Sometimes, asking:
"Can I get help under this government scheme?"
is much more natural than trying to figure out which website, menu, or form to open.
For Jan Sahay, the interaction is designed around:
Speak โ Understand โ Guide โ Take Action
The assistant can listen to the user, understand the request, retrieve relevant information when required, and respond naturally through voice.
The voice experience is powered by Murf Falcon, which I used for the text-to-speech side of the project.
๐งฉ The Main Features
๐๏ธ Government Scheme Guidance
Jan Sahay provides a dedicated area for government schemes and welfare information.
Users can ask questions about schemes and receive guidance around eligibility, benefits, and the information required to proceed.
The interface also has a dedicated Schemes Search section so users can explore scheme-related information beyond the voice conversation.
๐ก๏ธ Fraud Prevention
Digital fraud is another major focus of Jan Sahay.
The assistant is designed to help users understand common situations such as:
- UPI scams
- OTP-related fraud
- Phishing messages
- Suspicious links
- Digital payment safety
The goal is not simply to explain what a scam is.
It is to help users understand what they should do next.
๐ฐ Financial Literacy
Jan Sahay also focuses on making financial concepts easier to understand.
Instead of responding with complicated terminology, the assistant is designed to provide simple conversational guidance around financial topics.
This is particularly important when the user may not have a strong financial background.
๐ Complaint & Escalation Guidance
Sometimes an AI assistant should not try to solve everything itself.
Jan Sahay includes complaint and escalation capabilities so users can move toward appropriate human or official support when necessary.
The interface includes an Open Escalations section where escalation-related information can be managed.
This became one of the most important lessons I learned during the challenge:
A useful AI agent should know when it needs help.
๐ค From One Agent to Multiple Specialists
One of the features I am particularly happy with is the specialist-agent approach.
Instead of making one agent responsible for everything, Jan Sahay can work with specialized agents.
The current interface includes specialist options such as:
- ๐ก๏ธ Cyber Fraud Specialist
- ๐ Pension Specialist
- ๐ผ Business Loan Specialist
- ๐ฑ Crop Specialist
This allows the main assistant to act as the first point of contact and route a conversation toward a more focused specialist when required.
For example, a general question can stay with Jan Sahay.
But if a user asks a detailed question about a particular financial domain, the conversation can be handed to the appropriate specialist.
The user shouldn't have to restart the entire conversation just because a specialist is needed.
๐ง Building a More Context-Aware Agent
Another important part of the journey was adding memory.
A conversational agent becomes much more useful when it can retain appropriate context instead of treating every interaction as completely new.
This allows Jan Sahay to move beyond:
"Ask a question โ receive an answer."
toward:
"Understand the user โ remember useful context โ continue the conversation."
At the same time, financial applications require careful handling of sensitive information, so memory and escalation flows need to be designed with privacy in mind.
๐ Measuring the Voice Agent
By Day 8, I realized that simply saying "the agent works" isn't enough.
I needed a way to understand what was actually happening during conversations.
So I built a Call Performance Dashboard.
The dashboard tracks information such as:
- Total calls
- Successful calls
- Failed calls
- Call duration
- Language
- Channel
- Outcome
- Failure reason
- Recent call history
For example, during testing, my dashboard showed actual call records rather than hardcoded numbers.
This was useful because failed calls became something I could investigate instead of simply ignoring.
A voice agent shouldn't only be able to talk. It should be measurable and improvable.
๐ A Frontend Designed Around the Conversation
I also spent time improving the frontend experience.
The main voice interface clearly communicates the current state of the assistant:
Connecting โ Listening โ Speaking โ Call Ended
The interface also includes a live transcript so the user can see the conversation happening.
I added language-selection support and dedicated sections for:
- Home
- Schemes Search
- Fraud Prevention
- Complaint Helpline
- Open Escalations
- Call Dashboard
The idea was to make Jan Sahay feel like a complete citizen-assistance platform rather than just a microphone button connected to an AI model.
๐๏ธ How the System Works
At a high level, the voice interaction follows this flow:
๐ค USER
โ
โผ
๐๏ธ Voice Input
โ
โผ
Speech-to-Text
โ
โผ
๐ค Main AI Agent
โ
โโโโโโโโโโโผโโโโโโโโโโ
โ โ โ
โผ โผ โผ
Memory Tools Specialist
โ โ Agents
โ โ โ
โโโโโโโโโโโผโโโโโโโโโโ
โ
โผ
Murf Falcon
โ
โผ
๐ Voice Output
The important idea is that the LLM is not expected to do everything on its own.
The agent can use supporting components when the task requires them.
๐ My 10-Day Journey
Day 1 โ Getting Jan Sahay Talking
The first goal was simple: establish a working voice conversation.
I connected the voice pipeline and integrated Murf Falcon for speech generation.
Hearing the agent respond for the first time was a small moment, but it made the whole project feel real.
Day 2 โ Giving the Agent a Purpose
I moved from a generic voice assistant toward the Financial Services track.
Jan Sahay received a clear role focused on government schemes, financial literacy, fraud awareness, and citizen support.
Day 3 โ Personalising the Frontend
I worked on the frontend and made the experience more suitable for the users Jan Sahay is intended to serve.
The interface began to clearly communicate whether the agent was connecting, listening, or speaking.
Day 4 โ Adding Memory
I introduced memory so Jan Sahay could retain useful context and provide a more continuous experience across conversations.
Day 5 โ Connecting Real-World Data
The agent learned to use tools instead of relying only on what the language model already knew.
This was an important change:
Don't just generate an answer. Get the right information first.
Day 6 โ Making Outbound Calls
Jan Sahay gained outbound calling capability.
This opened up the possibility of an assistant that doesn't always wait for the citizen to initiate the conversation.
Day 7 โ Knowing When Humans Are Needed
I added human escalation.
The agent can recognize situations where human assistance is more appropriate and move the user toward the next step.
Day 8 โ Measuring Conversations
I built the call-performance dashboard and started looking at real conversation outcomes, failures, durations, and call history.
Day 9 โ Adding Specialist Agents
The main agent gained the ability to work with specialist agents for focused areas such as cyber fraud, pensions, business loans, and crop-related assistance.
This was one of the biggest steps toward making Jan Sahay a multi-agent voice system.
Day 10 โ Documenting the Journey
And now Day 10.
Instead of adding another feature, I am documenting what I learned and sharing the project so that someone else can understand how a voice agent can be built step by step.
๐งช What Was Difficult?
The most challenging part wasn't creating a single AI response.
It was getting all the pieces to work together.
A voice agent involves several moving parts:
- Real-time audio
- Speech recognition
- LLM reasoning
- Text-to-speech
- Frontend state
- Memory
- Tools
- Telephony
- Specialist agents
- Error handling
A problem in one part can affect the entire conversation.
I also spent time dealing with API configuration, model compatibility, voice integration, call flows, and making sure the different components continued working as the project became more complex.
The biggest lesson for me was:
Building an AI agent is less about one model and more about designing the entire system around it.
๐ Safety Matters
Because Jan Sahay deals with financial information, safety cannot be treated as an optional feature.
The agent should not encourage users to share sensitive information such as:
- Passwords
- OTPs
- PINs
- Card details
- Other confidential credentials
The system also needs to distinguish between general guidance and situations where the user should be directed toward official or human assistance.
That is especially important when building AI systems intended for real-world users.
๐ What I Learned From These 10 Days
Before starting this challenge, I mainly thought about voice AI as:
Speech โ AI โ Speech
After ten days, my understanding is very different.
A useful voice agent needs much more:
Voice + Context + Memory + Tools + Safety + Human Escalation + Specialist Agents + Analytics
The biggest lesson I am taking from this challenge is:
A good AI agent isn't one that tries to do everything. It's one that knows what to do, what not to do, and when to ask for help.
๐ฎ What's Next for Jan Sahay?
There is still a lot I would like to improve.
Some of the ideas I want to explore next include:
- More Indian language support
- Better regional-language conversations
- More verified government-scheme data
- More specialist agents
- Improved outbound calling
- Better call analytics
- Stronger privacy controls
- More robust production deployment
- Better accessibility for users with limited digital experience
The ten-day challenge may be over, but Jan Sahay doesn't have to be.
๐ Explore the Project
The complete project is available on GitHub:
Jan Sahay โ GitHub Repository
For the final DEV.to article, place your actual deployed/demo link here as well.
๐ Final Thoughts
Ten days ago, I started with the simple goal of making an AI voice agent talk.
Today, I have a system that can have voice conversations, remember context, use tools, make outbound calls, escalate to humans, track call outcomes, and hand conversations to specialist agents.
There were plenty of bugs, configuration problems, failed calls, and moments where something that looked simple turned out to be much harder than expected.
But that's probably the most valuable part of the experience.
I didn't just learn how to build a voice agent.
I learned how to design, debug, measure, and improve an AI system one step at a time.
Thank you to Murf AI for creating the 10 Days of Voice Agents โ VoiceForBharat Edition and giving me the opportunity to build and learn in public.
**Built with Murf Falcon for the voice experience
