Before this i had never registered for a challenge, or a hackathon. Not one. I just knew I liked AI, and one day I thought — let's actually try building something instead of just reading about it.
That decision turned into VyapaarMitra: a voice agent that can talk to customers on the phone, remember them, check a live catalogue, escalate to a human when needed, call people back on its own, and even hand off a conversation to a specialist when the question gets too specific for it to handle. Here's how it happened, and what it actually took to get there.
1. The problem, and who it's really for
Think about any small local shop — a kirana store, a boutique, a hardware shop — that manages its catalogue through a WhatsApp group or a shared PDF. It works, but only for customers who are comfortable scrolling through a WhatsApp catalogue, typing out questions, and waiting for a text reply. Not everyone is. Some customers would much rather just call and ask a person — "do you have this in stock?", "what's the price?", "is this available in blue?" — the way they've always shopped.
That's a real gap, and it's not a small one. Millions of small sellers in India run their business this way, and a chunk of their customers are quietly opting out simply because typing isn't how they want to shop.
2. What VyapaarMitra actually does
VyapaarMitra is a voice agent built for the Local Commerce track of Murf AI's "10 Days of AI Voice Agents" challenge. A customer calls a number and talks to it exactly like they'd talk to a shop assistant on the phone. It answers questions from a live catalogue — price, stock, ratings — in Telugu, Hindi, English, or a natural mix of all three. If the customer wants to order, they say "confirm order," and the agent doesn't try to close the sale itself — it escalates that request straight to the seller, who ships it.
That last part matters. The agent isn't trying to replace the seller. It's clearing the one step that stops a lot of customers before they even reach the seller: "does this shop even have what I want?" Voice makes sense here specifically because it removes the friction of typing, reading a catalogue, and composing a message — for exactly the customers who'd rather just talk.
3. What I actually built, and why each piece mattered
A voice that doesn't sound like a robot. The agent speaks using Murf Falcon — an Indian voice (Anisha) that sounds natural on a phone call, not like a call-center IVR reading a script.
A personality with boundaries. Before anything else, I had to give the agent a clear identity: what it's allowed to say, what it's never allowed to do. It can't make false claims about stock or pricing, and it can't confirm an order on its own authority — that always goes to a human. Without this, a voice agent is just a confident liar with good pronunciation.
Conversations that don't stay in one language. Real customers here don't speak in clean, single-language sentences — they code-switch mid-sentence between Telugu, Hindi, and English, and the agent had to hold up in that mess, not just in a scripted single language.
A frontend that shows what's happening. Since a phone call has no visuals by default, I built a small interface showing the agent's state in real time — ready, listening, speaking, connecting, call ended — so during testing (and demos) it's obvious what the agent is doing at any moment.
Memory, but only with permission. Returning callers get remembered — but only if they agree to it. No consent, nothing gets saved. That felt non-negotiable once I thought about it: a voice agent overhearing and quietly storing everything a caller says is not something I wanted to build, even for a challenge.
A live catalogue it can actually check. The agent pulls real price, rating, and stock data instead of guessing. Small detail that turned into a real fix: the catalogue API returned prices in dollars, which meant nothing to a customer expecting rupees — so I had to convert every price before the agent could say it out loud.
Outbound calls — the agent calling out, not just answering. This was the feature that made the whole thing feel real. The agent can dial a customer and deliver an order confirmation on its own, using LiveKit's SIP integration.
Knowing when to bring in a human. For anything sensitive — a payment dispute, an order problem, anything outside its depth — the agent doesn't guess. It raises an alert with a short summary, an urgency level, and a reference ID, so a real person can step in fast.
A dashboard that shows the numbers. Total calls, successful calls, failed calls, success rate, all on one analytics tab — because a voice agent you can't measure is a voice agent you can't improve.
A specialist that takes over mid-call. When a caller asks about a return or refund, the main agent hands the conversation to a dedicated Returns & Refunds specialist — a different voice (Samar, deliberately more distinct from Anisha), and the interface itself shifts to a different color theme so it's visually obvious a different "person" has picked up the conversation.
4. The two problems that actually broke me
Everything above sounds clean in a list. It wasn't.
The first one hit right after I got outbound calling working. The call would connect — the phone would ring, I'd pick up — and then nothing. Dead silence from the agent. No error, no crash, just a connected call with nobody talking. I spent hours on this one, convinced something was wrong with the SIP setup or the audio pipeline. It wasn't. The actual bug was that my "outbound greeting" logic wasn't scoped to only outbound calls — it was tangled into the general session logic, so it silently short-circuited the normal reply flow instead of triggering cleanly. The fix, once I found it, was almost embarrassingly small: separate the outbound-only logic from everything else.
The second one showed up right after I added the specialist agent. I asked it a return-related question, expecting the handoff to kick in — instead, it fired off a human escalation, like the specialist didn't exist at all. It took me a moment to realize the problem wasn't the handoff code — it was that my escalation logic had no concept of "this belongs to the specialist now." I'd only ever taught the agent to route order problems to a human; nothing told it that return and refund questions had a specialist of their own. I rewrote the system prompt to be explicit about all three paths — catalogue questions stay with the main agent, order confirmations go to a human, return and refund questions go to the specialist — and only then did the handoff actually behave the way I expected.
Neither bug was exotic. Both were the same lesson wearing different clothes: an LLM will always do something when it's unsure — it just won't necessarily do the right thing unless you've told it, very precisely, what "right" means in that exact situation.
5. If you want to build one yourself
A voice agent like this comes down to four moving pieces:
- Speech-to-Text — turns what the caller says into text the system can read
- An LLM — decides what to say back, and which tool or action to trigger
- Text-to-Speech — turns the reply into natural-sounding audio
- Real-time transport — carries the audio both directions, including over an actual phone line
I built this on top of the Murf LiveKit Starter template — it's a solid base if you want to start from something working rather than from zero. Clone it, install dependencies, and set up a local environment file for your API keys. Keep that file out of version control entirely — never commit real keys, phone numbers, or any caller data to a public repo. You'll need keys for your speech-to-text provider, your LLM, your TTS provider, and your real-time transport layer.
Once it's running, you can talk to the agent directly from a browser to test the conversation flow, and layer in phone connectivity once the core logic works the way you want.
6. What I'd improve next
More specialist agents — right now there's only one (Returns & Refunds). Adding more (payments, delivery tracking) would make the handoff logic more scalable.
Testing against a real seller's catalogue — so far I've used a sample/test catalogue. Integrating with an actual local shop's live inventory and testing with real customers is the natural next step.
Two-way escalation tracking — right now the agent sends an alert to Discord but never learns whether the seller responded or the order was confirmed. Closing that loop would let the agent follow up with the caller too.
Better error handling for outbound calls — to catch issues like the Day 6 silent-call bug before they happen, with retry logic or a fallback message if a call fails.
A richer analytics dashboard — beyond total/success/fail counts, adding average call duration, common query types, or a breakdown of escalation reasons would give the seller more useful insight.
7. Links
Ten days ago I hadn't finished a single project. Now I've got a voice agent that calls people, remembers them (with permission), and knows when to get a human involved. That, more than anything else, is the actual takeaway from this challenge.
Built during "10 Days of AI Voice Agents — VoiceForBharat Edition" by Murf AI, using Murf Falcon, the fastest TTS API. #VoiceForBharat
