From a simple voice assistant to a multi-agent disaster-response system
Disasters don't wait for people to open an app, read instructions, or type a message.
In an emergency, people need information quickly and sometimes they simply need to talk.
That was the idea behind Ashraya AI, the disaster-response voice agent I built during the 10 Days of Voice Agents, VoiceForBharat Edition challenge by Murf AI.
Over the past 10 days, I went from building a basic conversational voice agent to developing a system with memory, tools, outbound calling, call analytics, success tracking, human escalation, and specialist-agent handoffs.
The goal was not to build another chatbot that can answer everything.
The goal was to build a voice-first system that can guide people during disaster situations while being explicit about what it does not know.
1. The Problem
During disasters such as floods, earthquakes, cyclones, or other emergencies, information can become difficult to access.
A person may need answers to questions like:
What should I do right now?
What should I take if I need to leave?
Where can I get help?
How should I prepare an elderly family member for evacuation?
Can someone help me if I cannot handle the situation myself?
A traditional text interface isn't always ideal in these situations.
Someone may be frightened, busy helping family members, unable to type comfortably, or simply need to communicate naturally.
That's where a voice interface can be useful.
Ashraya AI is designed as a conversational disaster-response assistant that users can interact with through voice.
The project focuses on providing practical guidance while maintaining strict safety boundaries.
2. Why Voice?
Voice changes the interaction model.
Instead of navigating menus or typing questions, a user can simply say:
"There is flooding near my house. What should I do?"
or:
"I may need to go to a shelter. What should I take?"
The agent can respond conversationally and ask relevant follow-up questions.
For a disaster-response use case, this matters because the user's attention may already be occupied by the emergency itself.
But voice introduces another challenge:
An AI that sounds confident can easily sound authoritative—even when it is wrong.
That made safety and hallucination prevention one of the most important parts of this project.
3. What is Ashraya AI?
Ashraya AI is a voice-based disaster-response assistant built using:
Murf Falcon — text-to-speech
LiveKit Agents — real-time voice-agent infrastructure
Deepgram — speech-to-text
Gemini — language model
Python — application logic and agent orchestration
The architecture evolved throughout the challenge.
Instead of treating Ashraya as one giant agent that handles every possible question, I eventually introduced a specialist-agent architecture.
The main agent handles the overall disaster-response conversation.
When the conversation becomes specifically shelter-related, it can hand the conversation to a Shelter Information Specialist.
This creates a much clearer separation of responsibilities.
4. System Architecture
At a high level, the voice interaction works like this:
User
│
▼
Microphone
│
▼
Speech-to-Text
(Deepgram)
│
▼
Main AI Agent
(Gemini)
│
┌───────────┴───────────┐
│ │
▼ ▼
General disaster Shelter-related
assistance request
│
▼
Shelter Information
Specialist
│
▼
Shelter Guidance
(without fabricated
live information)
│
▼
Murf Falcon TTS
│
▼
User
The important design decision here is that the specialist is not simply another prompt inside the main agent.
It has its own instructions, scope, limitations, and success tracking.
5. Building the Voice Agent
A real-time voice agent fundamentally needs several components working together:
Speech-to-Text
The user's speech needs to be converted into text so the language model can understand it.
I used Deepgram for speech recognition.
Language Model
The language model handles reasoning, conversation, tool selection, and responses.
I used Gemini for this layer.
Text-to-Speech
The generated response needs to be converted back into natural speech.
For this project, I used Murf Falcon.
Real-Time Transport
The system needs a real-time communication layer to move audio between the user and the agent.
I used LiveKit Agents.
Together, these components form the basic voice-agent pipeline:
User Speech
↓
Speech-to-Text
↓
LLM / Agent Logic
↓
Text-to-Speech
↓
Spoken Response
6. Giving the Agent a Clear Personality and Safety Boundary
One of the first lessons I learned was that an agent's prompt is not just about telling it what to say.
It also needs to define:
What it is allowed to do
What it is not allowed to do
When it should ask questions
When it should use a tool
When it should escalate
What information it must never invent
For a disaster-response system, this becomes especially important.
For example, an unsafe agent might respond to:
"Which government shelter is open near me?"
with a fabricated answer such as:
"The nearest shelter is XYZ School."
That would be unacceptable.
Ashraya AI is explicitly instructed not to invent live shelter locations, availability, capacity, evacuation routes, or government orders when it doesn't have verified real-time data.
Instead, it should clearly state the limitation and direct the caller toward appropriate official local authorities.
That distinction became one of the most important safety principles in the project:
Not knowing something is better than confidently making something up.
7. Memory for Returning Users
Another part of the project was giving Ashraya AI memory.
The idea was to avoid treating every conversation as if it were happening with a completely new user.
Memory can allow the system to retain useful conversational context and provide more consistent interactions when appropriate.
However, memory in a disaster-response system also needs to be handled carefully.
The agent should use relevant context without making assumptions about the user's current emergency.
Current information from the caller should always take priority over stale conversational context.
8. Tools and Actions
The agent is not limited to generating text.
Tools allow it to perform specific actions.
This is important because an AI agent should not rely on the language model to simulate actions that should actually happen in software.
For example, instead of simply saying:
"I'll connect you to a specialist."
the system can invoke a handoff tool that actually transfers the conversation.
This became particularly important during Day 9.
9. Multi-Agent Handoff: The Shelter Specialist
One of the biggest architectural changes was introducing a dedicated Shelter Information Specialist.
The main Ashraya AI agent is broader.
The shelter specialist is intentionally narrow.
It focuses on questions such as:
What should I take to a shelter?
How should I prepare before moving?
What documents should I carry?
What medicines or essential supplies should I take?
What should I consider when travelling with children or older adults?
What information should I obtain from local authorities?
But it is not a general disaster-response agent.
If someone asks the specialist:
"What should I do during an earthquake?"
or:
"How do I perform CPR?"
the specialist should not suddenly become a medical or earthquake-response expert.
It should say that the question is outside its role and redirect the user appropriately.
This is a subtle but important difference between a multi-agent system and a single agent with a very large prompt.
The goal isn't:
"Make every agent know everything."
The goal is:
"Give each agent a clear responsibility and let the system decide who should handle the conversation."
10. Preventing Hallucinated Shelter Information
This was one of the most important safety challenges in the project.
Suppose someone asks:
"Which government shelter is open near me right now?"
If the system doesn't have a verified live shelter database, it cannot honestly answer that question.
So the Shelter Information Specialist explicitly states that it does not have verified real-time shelter-location or opening-status information.
It can instead explain how the caller can obtain verified information from appropriate local authorities.
This is an important design principle for real-world AI systems:
The absence of information should produce a limitation—not a hallucination.
A disaster-response assistant should be conservative when dealing with information that can directly affect someone's physical safety.
11. Outbound Calling
Ashraya AI was also designed to support outbound voice interactions.
This changes the experience from:
"A user opens an AI application."
to:
"The AI can initiate a voice conversation."
For a disaster-response scenario, outbound communication could eventually be useful for alerting people, checking whether they need assistance, or providing information.
However, outbound calling also introduces additional considerations around consent, privacy, call reliability, and responsible use.
12. Human Escalation
Ashraya AI is designed to assist callers during disaster situations, but it is not intended to replace human support.
I added the ability for Ashraya AI to identify situations where human assistance may be required and route the conversation toward an escalation workflow.
The system does not pretend that a human has been contacted unless the handoff actually occurs.
This was an important design principle throughout the project:
A reliable disaster-response AI should know its limits.
The goal is not to make the AI handle every situation on its own. Instead, it should recognize when human assistance may be more appropriate and follow the appropriate escalation path.
The agent should also never claim that a human operator, rescue team, or emergency service has been contacted unless that action has actually taken place.
13. Call Analytics and Success Tracking
Another lesson from the project was that a conversation isn't necessarily successful simply because the AI produced a response.
I added call outcome tracking so conversations can be evaluated based on whether the intended objective was actually completed.
For example, a shelter-related interaction should only be marked successful after meaningful shelter guidance has actually been provided.
A transfer alone should not automatically count as success.
This distinction is important for evaluating voice agents in production.
You need to measure outcomes, not just conversations.
14. The Hardest Part: Making the Agent Stay in Its Lane
One of the biggest challenges I encountered was getting the specialist-agent architecture to behave correctly.
At one point, the Shelter Information Specialist was still answering questions about earthquakes and CPR.
That was a clear architectural failure.
The specialist had been created, but its boundaries were not being enforced strongly enough.
I tightened its instructions so that it explicitly refuses questions outside its shelter-information scope.
This taught me an important lesson:
Creating a specialist agent isn't enough. You have to define its boundaries as carefully as its capabilities.
15. When a Successful Call Was Marked as a Failure
Another challenge I faced was with call outcome tracking.
During testing, I had conversations where the agent successfully handled the user's request, but the call was still marked as failed in the call analytics.
At first, I wondered whether the caller needed to explicitly say something like "okay" or "thank you" before ending the conversation.
But that wasn't the real issue.
I realized that a conversation ending and a task being successfully completed are not necessarily the same thing.
For Ashraya AI, a successful call should depend on whether the intended objective was actually achieved—not simply on whether the caller stayed on the line or how the conversation ended.
This led me to add explicit success tracking for the relevant agent workflows, including the shelter specialist.
That experience taught me an important lesson about voice agents:
Don't use the end of a conversation as a substitute for measuring its outcome.
For a real disaster-response system, reliable outcome tracking is just as important as generating a good response.
16. Another Important Lesson: "Transfer" Must Actually Transfer
One of the most interesting things I learned was that saying:
"I'm connecting you to the Shelter Information Specialist."
doesn't mean the transfer actually happened.
The application needs to:
Detect the appropriate intent.
Invoke the transfer tool.
Create or activate the specialist agent.
Pass the relevant conversation context.
Return control to the specialist.
Ensure the specialist follows its own instructions.
If any of these steps fail, the main agent may continue speaking.
That creates a confusing experience where the user asks:
"Who am I speaking to?"
and the system gives inconsistent answers.
This was a useful reminder that agent handoffs are software orchestration problems as much as they are prompting problems.
- How to Run the Project
The project is built using Python, LiveKit, Murf Falcon, Deepgram, and Gemini, with separate backend and frontend components.
Step 1: Clone the Repository
git clone https://github.com/AsmiRaut23/murf-livekit-starter
cd murf-livekit-starter
Step 2: Backend Configuration
Create a .env.local file inside the backend directory:
backend/.env.local
Add the required backend credentials:
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_gemini_api_key
These credentials allow the backend to connect to LiveKit and use the speech and language services required by Ashraya AI.
Step 3: Frontend Configuration
Create a .env.local file inside the frontend directory:
frontend/.env.local
Configure the LiveKit connection:
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
The frontend uses these values to connect to the LiveKit environment and communicate with the voice agent.
Step 4: Install Backend Dependencies
Navigate to the backend:
cd backend
Create a virtual environment:
python -m venv .venv
Activate it on Windows:
.venv\Scripts\activate
Install the dependencies:
pip install -r requirements.txt
Step 5: Run the Agent
Start the Python LiveKit agent using the project's configured entry point.
The frontend can then be started separately and connected to the LiveKit room to test Ashraya AI through a voice conversation.
18. Testing the Agent
The easiest way to test a voice agent is through real conversations rather than only checking whether the code runs.
For Ashraya AI, I tested scenarios such as:
General disaster question
"There is flooding near my house. What should I do?"
Shelter question
"I may need to go to a shelter. What should I take?"
Live-information limitation
"Which government shelter is open near me right now?"
Specialist boundary
After transferring to the Shelter Information Specialist:
"What should I take to the shelter?"
and then testing an unrelated question such as:
"How do I perform CPR?"
The specialist should not start behaving like a general medical assistant.
Testing these boundary cases was just as important as testing successful conversations.
19. What I Would Improve Next
Ashraya AI is still a prototype, and there are several areas I would improve before treating it as a production disaster-response system.
Verified real-time shelter data
The biggest improvement would be connecting the shelter specialist to an authoritative, continuously updated government or disaster-management data source.
That would allow the system to answer location-specific shelter questions based on verified information rather than simply explaining that live data is unavailable.
Better emergency escalation
A production version would need carefully designed integration with emergency services and human operators.
More robust multilingual support
India has enormous linguistic diversity.
Expanding reliable multilingual and code-mixed conversations would make the system more accessible.
Stronger evaluation
I would also build a larger evaluation suite covering:
Hallucination resistance
Specialist boundaries
Emergency scenarios
Tool failures
Handoff failures
Call completion
Latency
False transfers
Successful outcomes
20. What I Learned in 10 Days
The biggest lesson from this challenge wasn't how to connect speech-to-text to an LLM and then to text-to-speech.
It was learning that a useful voice agent is much more than a model that can talk.
A reliable system needs:
Conversation + tools + memory + guardrails + orchestration + observability + failure handling.
And sometimes the most important behavior of an AI agent is knowing when not to answer.
Building Ashraya AI also changed how I think about multi-agent systems.
I initially thought the challenge was to make one agent increasingly capable.
By the end, I realized that a better approach is often to build a system of focused agents with clear responsibilities.
Final Thoughts
Over these 10 days, Ashraya AI evolved from a voice assistant into a more structured disaster-response system.
It can:
Have natural voice conversations
Use an Indian voice powered by Murf Falcon
Follow explicit safety guardrails
Maintain conversational context
Use tools
Support outbound calling
Track call outcomes
Escalate when necessary
Hand shelter-related conversations to a specialist agent
Refuse to fabricate information it cannot verify
The project is still a work in progress.
And that's probably the most valuable part of the experience.
Building a voice agent isn't about making AI sound impressive.
It's about making the system useful, predictable, and honest about its limitations.
That's the direction I want to continue exploring.
Tech Stack
Murf Falcon — Text-to-Speech
LiveKit Agents — Real-time voice infrastructure
Deepgram — Speech-to-Text
Gemini — LLM
Python — Agent logic and orchestration
Project Links
🔗 GitHub: https://github.com/AsmiRaut23/murf-livekit-starter
💼 LinkedIn: www.linkedin.com/in/asmiraut
