I once bought a share at about ₹6 without understanding what the complete transaction would cost. I later sold it around the same price and was surprised to see a loss of roughly ₹50 somewhere in my account. I did not know whether that number came from the contract note, ledger, available funds or P&L view. The important lesson was not that every small trade has the same charge. It was that I had acted without understanding brokerage, taxes or where a number in a broker app actually came from.
That experience became the starting point for FinEd Saathi, the voice-first financial literacy tutor I built during 10 Days of Voice Agents - VoiceForBharat Edition. I chose the Financial Services track because I wanted to solve a problem I had felt myself. The challenge listed Financial Services and Learning & Literacy as two separate tracks. I formally stayed in Financial Services, but I deliberately merged finance with education in the product because access without understanding was the problem I wanted to solve. The tutor is for beginners who want Indian market concepts explained in a patient conversation before they put real money at risk. The public code is in the FinEd Saathi repository.
The problem I wanted to solve
Financial education often assumes that a learner already knows the vocabulary. A person who is still asking what an ETF is may be sent to a fee schedule, a tax circular or a dense product page. Even correct information can be hard to use when several unfamiliar concepts arrive at once.
I wanted the product to begin where the learner is. FinEd can explain stocks, mutual funds and SIPs, ETFs, gold, F&O, IPOs and bonds. It can also unpack a confusing charge without inventing a reason. In my ₹6 share story, the agent treats the price profit or loss as zero until I show where the separate loss appeared, such as a contract note, ledger or P&L view. That is more useful than confidently guessing.
The landing page lets a beginner choose a topic before starting a voice lesson.
Why voice changed the product
Voice makes the lesson feel like a conversation instead of an exam. A learner can ask a short question, interrupt or switch between English, Hindi and a code-mixed register. Deepgram Nova-3 handles multilingual speech recognition. Gemini handles the teaching conversation and tools. LiveKit carries the real-time session.
Murf Falcon is the fastest TTS API. I use Murf Falcon 2 with Nikhil as the FinEd voice because his Indian conversational delivery fits a patient tutor. Murf documents Falcon 2 as a real-time speech model with roughly 100 ms time to first audio. The low-latency path matters because a financial explanation quickly feels unnatural when every turn contains a long pause.
The voice workspace shows what the agent heard, which specialist is active and what the learner can do next.
How FinEd Saathi works
The browser sends audio through LiveKit to a Python worker. Deepgram transcribes the speech, Gemini produces a bounded response and Murf Falcon 2 streams the spoken answer. FinEd owns ordinary lessons. TaxEd is a separate specialist reached only after permission. Optional market data, memory, paper trading, human help, outbound telephony and analytics sit outside the essential voice path so missing optional credentials do not break the core tutor.
The main voice session is deliberately compact:
fined_tts = murf.TTS(
voice="Nikhil",
style="Conversational",
model="falcon-2",
locale="en-IN", # Initial locale, selected again before each reply stream.
tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),
text_pacing=True,
)
fined_locale = FinEdTTSLocaleController(fined_tts)
session = AgentSession[SessionState](
userdata=state,
stt=fined_stt,
llm=fined_llm,
tts=fined_tts,
turn_detection=inference.TurnDetector(version="v1-mini"),
vad=ctx.proc.userdata["vad"],
preemptive_generation=True,
)
Before LiveKit opens each FinEd speech stream, the controller peeks at the actual reply. Devanagari Hindi selects hi-IN; English and Latin-script code-mixed replies select en-IN. That keeps Nikhil fixed while applying a supported locale before Murf snapshots the stream options. TaxEd creates a separate Murf TTS instance using Anusha and a server-normalized en-IN, hi-IN or hi-LATN locale.
What the learner can actually do
A learner can ask for definitions, compare concepts or request a read-only market quote when optional Angel One access is configured. Quote tools can search instruments, return current prices and fetch historical daily closes. They cannot read holdings, positions or account information. If the broker token is missing or expired, the agent says that live data is unavailable instead of guessing a price.
The paper portfolio starts with ₹1,00,000 in virtual cash. It supports simulated NSE EQ cash equity or ETF delivery orders based on a fresh quote. The browser prepares a draft then requires explicit confirmation of that same unexpired draft. The portfolio, virtual cash and fill history stay in browser storage. No real broker order API is called.
Paper trading gives the learner a safe place to practise without connecting a real trading account.
Sessions behave more like a familiar chat product. Meaningful transcripts are stored under local session identifiers in the browser so the learner can switch conversations. Caller memory is separate, private and consent-gated. It saves only learning preferences after a fresh yes. Credentials, government identifiers, account numbers, holdings, income and bank details are excluded.
The analytics page reports anonymous totals such as call count, speaking time, committed handoffs and successful calls. It stores no audio, transcript, utterance text, caller identity or phone number.
Analytics proves whether the experience completed useful actions without turning learners into a surveillance dataset.
Why I built TaxEd as a separate specialist
Investment tax questions need a stricter evidence boundary than a general concept lesson. FinEd first asks whether the learner wants to connect. Only a fresh explicit yes transfers the tax question to TaxEd. Returning to FinEd also requires permission.
TaxEd speaks with Anusha, a Murf Indian voice whose official voice-library entry supports English and Hindi. It searches a packaged registry of official Indian tax rules, states the relevant applicability date and links the source. A rule must be current for the requested date. When a current packaged source is missing, uncertain or past its review date, TaxEd abstains. It does not calculate a person's final liability, file a return or help evade tax.
The handoff is visible, consented and evidence-first.
Safety was part of the architecture
FinEd is education, not investment advice. Deterministic guardrails run before provider inference for real trading, personalized recommendations, guaranteed outcomes, unsafe F&O calls, credential requests and tax evasion. A request to place a real order is refused even when it also mentions paper trading.
The system never asks for a broker password, PIN, OTP, PAN, Aadhaar, bank detail or full account number. F&O mode teaches mechanics, payoff examples and risk only. Human-help requests use a short redacted summary and require fresh consent before storage. Outbound calling is an operator-only optional path that also requires explicit consent. Missing telephony configuration leaves browser voice available and makes the call fail closed.
I documented the deterministic evidence in RED_TEAM.md, including broker identifier redaction during a TaxEd handoff, refusal before model inference and abstention on an unverified tax rule.
The hardest parts
The hardest work was not drawing the interface. It was making state transitions reliable. A handoff that only changes a badge is not a handoff. I had to keep the active specialist, the TTS voice, the transferred question and the return path aligned. I also had to stop repeated permission loops when a clear yes had already been given.
Multilingual speech exposed a similar issue. A multilingual voice identity is not enough when the synthesis locale is fixed. FinEd now routes Nikhil to hi-IN for Devanagari Hindi and en-IN for English before audio begins. TaxEd keeps Anusha while a server-normalized en-IN, hi-IN or hi-LATN locale follows the handoff language. The browser cannot choose arbitrary synthesis settings.
Live market data taught me to design for absence. Broker access tokens expire. The correct fallback is not a stale or invented price. The agent now says the quote is unavailable while concept lessons, the empty paper dashboard and other safe features continue.
I also learned that proof assets are part of product quality. A blurred screenshot can hide the very behavior the article claims to demonstrate. I recaptured the specialist handoff and analytics views so the state, source and privacy boundary are readable.
Run it yourself
You need Python 3.10 through 3.14, uv, Node.js and the repository's pinned pnpm 9 release. Clone the public repository then create local environment files:
cp backend/.env.example backend/.env.local
cp frontend/.env.example frontend/.env.local
Add these required backend values to backend/.env.local:
LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY
Use the same LiveKit project values in frontend/.env.local and keep the worker name:
LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
AGENT_NAME=my-agent
Keep real values only in .env.local. Never commit them. Install the backend and its local voice models:
cd backend
uv sync
uv run -m livekit.agents download-files
uv run dotenv -f .env.local run -- python src/agent.py start
In a fresh shell, install and run the frontend:
cd frontend
pnpm install
pnpm dev --port 3001
Open http://127.0.0.1:3001, select a learning mode, choose Talk to FinEd Saathi and allow microphone access and browser audio playback. Ask "What is an ETF?" Then ask how an equity ETF is taxed and say "Yes, connect me to TaxEd" when FinEd requests permission.
Common fixes:
- Connection fails: confirm frontend and worker use the same LiveKit project and
AGENT_NAME. - Voice is silent: confirm
MURF_API_KEY, microphone permission and browser audio playback. - Live quote unavailable: refresh the optional Angel One access token. Do not guess a quote.
- TaxEd abstains: verify the packaged official rule is current for the requested date.
- Outbound call fails: verify explicit consent, the stored LiveKit SIP trunk and Twilio termination setup.
Angel One, Twilio, the local knowledge index and outbound calling are optional. A normal voice lesson works without them.
How I tested it
The backend has a deterministic suite for guardrails, tools, handoffs, tax rules, memory, analytics and outbound consent. The frontend contracts cover the public interface, browser paper portfolio, token route and documentation. I keep the provider-backed evaluation separate because it requires valid external credentials and inference access.
cd backend
uv run pytest -q --ignore=tests/test_agent.py
uv run ruff check .
uv run ruff format --check .
cd frontend
node --test tests/*.test.mjs
pnpm exec tsc --noEmit
pnpm format:check
pnpm build
The red-team record states exactly which deterministic tests support each claim. I prefer that to describing a manual demo as proof of every failure path.
What I would build next
I would make the source registry easier to update with a review workflow, add more broker-independent historical lessons and test interruption latency across a wider set of Indian network conditions. I would also add an export that lets a learner keep a private summary of concepts they understood without exporting a transcript.
Ten days turned one confusing small trade into a product I personally find smooth to use. More importantly, it changed how I think about a finance assistant. The best version is not the one that sounds most confident. It is the one that teaches clearly, proves its sources, asks permission at boundaries and knows when to stop.
