Building Dukaan Mitra: A Voice AI Assistant for Kirana Stores

Over the last 10 days, I took part in 10 Days of Voice Agents — VoiceForBharat Edition, where the goal was to build a capable voice AI agent from scratch. Here's a look at what I built, the challenges I ran into, and how you can build one too.

The Problem and the Users

Local grocery shops, known as "Kirana" stores in India, run on personal relationships with regular customers. A large share of their business comes from customers calling in their monthly restock of staples like rice, dal, and oil.

But during peak hours, shop owners miss these calls or forget to follow up — and that means lost sales. I wanted to fix this by building Dukaan Mitra, an AI voice assistant that proactively calls regular customers to remind them about their monthly restock, takes their order, and saves it directly to a database.

What Dukaan Mitra Does

Dukaan Mitra is designed to sound and act like a real shop assistant:

It makes outbound calls to customers and greets them by name.
It understands natural, code-mixed conversation — Hinglish, Hindi, and English.
It checks today's prices and stock for items in real time.
It confirms the order, generates an Order ID, and logs it.
If a customer raises a complaint, dispute, or return request, it hands the call over seamlessly to a specialist agent named Pooja.

The Most Important Features

A few key features are what made this agent feel robust and realistic rather than a scripted demo.

1. Indian voices with Murf Falcon
To make the agent sound local, I used Murf Falcon for text-to-speech — the main assistant runs on the voice "Anisha," and the returns specialist runs on "Pooja." Falcon's sub-130ms latency is what made the conversation feel genuinely responsive instead of laggy and robotic.

2. Deep guardrails and personality
I instructed the LLM to strictly use female grammatical forms when speaking Hindi/Hinglish (e.g., "main kar dungi," not "main kar dunga"). I also gave it a hard rule to never invent prices — if an item isn't in the database, the agent says so honestly instead of guessing.

3. Real-time tools and memory
The agent connects to local SQLite databases (inventory.db and users.db) and uses tools like check_price_and_stock, confirm_order, and save_caller to interact with that data mid-call.

4. Multi-agent handoff
Returns are tricky to handle well in one flow. When a customer asks for a refund, Dukaan Mitra calls a transfer_to_returns_specialist tool that routes the conversation to a completely separate agent instance — different prompt, different voice — dedicated entirely to logging return requests.

The Difficult Parts: Lessons Learned

Building this wasn't without hurdles. The biggest one was a race condition right at the start of every outbound call.

Since the agent needs to know who it's calling, I initially had it call the lookup_caller tool as its very first action. But I discovered that Gemini 3.5 Flash Lite would sometimes crash or hang if it tried to invoke a tool as its absolute first action, before any conversational turns had happened.

How I fixed it: I changed the backend initialization flow. Instead of letting the agent look up the user itself, I wrote a Python script to pre-fetch the caller's information from SQLite before the agent even initializes, then injected that data directly into the system prompt as CURRENT CALLER INFO. This removed the need for an initial tool call entirely — it stopped the crashes and also made the agent's first response noticeably faster.

How to Build and Run Your Own

If you want to build something similar, here are the main components:

Speech-to-Text (STT): Deepgram — turns user speech into text
LLM: Google Gemini — the brain of the agent
Text-to-Speech (TTS): Murf Falcon — turns text back into voice
Real-Time Transport: LiveKit — handles the WebRTC audio streaming

Prerequisites:

Python 3.10+ (with uv for fast package management)
Node.js 18+ (with pnpm)
A free LiveKit Cloud account and project

Step 1: Clone the repository

git clone https://github.com/gauravshinde2162006-hash/Day-1-Get-Your-Voice-Agent-Talking.git
cd Day-1-Get-Your-Voice-Agent-Talking

Step 2: Add your API keys

Create a .env.local file in both the backend/ and frontend/ directories with the following:

LIVEKIT_URL=wss://...
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_secret
MURF_API_KEY=your_murf_key
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_gemini_key

.env.local is already in .gitignore — never commit real keys to the repo.

Step 3: Install dependencies and run

In one terminal, start the backend agent:

cd backend
uv sync
uv run python src/agent.py dev

In a second terminal, start the frontend:

cd frontend
pnpm install
pnpm dev

Step 4: Test a conversation

Open http://localhost:3000, click Start talking, and allow microphone access. Since this simulates an outbound call, Dukaan Mitra greets you first — just like it would greet a real customer.

What I'd Improve Next

Dukaan Mitra works well as a proof of concept, but a few rough edges are honestly still there:

Hardcoded caller identification. The backend currently initializes each session with a fixed caller_id. In production, this should come dynamically from the incoming SIP trunk data or the frontend session instead.
Mocked WhatsApp integration. The send_whatsapp_message tool doesn't actually send anything in the background yet — it opens a pre-filled WhatsApp Web tab via webbrowser.open(). Swapping this for a real Twilio or WhatsApp Business API call is next on the list.
Simulated error handling. To test the agent's failure guardrails during a database outage, I currently check for a local force_db_fail.flag file. A real version needs proper HTTP timeout and try/except handling against a live inventory API.