I Built an AI Voice Sales Agent — Here’s the Architecture Behind It

I Built an AI Voice Sales Agent — Here’s the Architecture Behind It

What if a dealer could simply call a number, tell an AI what they need, and place an order through a normal conversation?

No app.
No searching through products.
No filling out forms.

Just talk.

That was the idea behind a project I've been working on: an AI Voice Sales Agent for dealers.

For example:

"I need 50 boxes of Product X and 20 of Product Y."

The AI should understand the request, check whether the products are available, figure out the dealer's pricing, apply any applicable schemes, confirm the order, and eventually push it into the company's ERP.

This isn't a finished enterprise product yet. I'm still building and figuring things out, but I wanted to document how I'm approaching the architecture and some of the decisions I've made along the way.

The Architecture

The basic flow looks something like this:

Dealer
↓
Phone Call
↓
Twilio
↓
OpenAI Realtime API
↓
LangGraph
↓
Business Tools
↓
FastAPI Backend
↓
PostgreSQL / Redis
↓
ERP

The interesting part isn't just getting an LLM to talk.

The AI actually needs to do things.

It should be able to:

• Understand what the dealer wants
• Search for products
• Check inventory
• Get dealer-specific pricing
• Check applicable schemes
• Remember useful customer context
• Create draft orders
• Eventually sync with an ERP
• Handle failures without making things up

Let's break down how I'm thinking about each part.

1. The Voice Layer

The first challenge is simple:

How does the dealer actually talk to the system?

I'm using Twilio as the telephony layer.

The basic flow is:

Dealer calls
↓
Twilio receives the call
↓
Audio goes to the AI system
↓
AI processes the conversation
↓
Response is generated
↓
Dealer hears the response

Initially, I thought of voice as simply:

Speech → Text → LLM → Text → Speech

But once you start thinking about an actual conversation, latency becomes a huge deal.

Imagine saying something to an AI and waiting 4–5 seconds for every response.

Technically, it works.

As a conversation?

Not great.

That's why I'm looking at real-time voice capabilities rather than treating the system like a normal chatbot with a microphone attached to it.

1. The AI Layer

For the conversational intelligence, I'm using the OpenAI Realtime API.

But here's something I realized pretty quickly:

The LLM shouldn't be responsible for everything.

For example, if a dealer says:

"Give me 50 of the blue ones."

The AI needs to understand what "blue ones" refers to.

That requires conversation context.

The system might have something like:

Dealer:
ABC Distributors

Current conversation:
Product: Product X
Variant: Blue
Requested quantity: 50

Dealer preferences:
Preferred warehouse: Pune
Preferred language: English

So the AI understands the conversation instead of treating every sentence as an isolated question.

1. LangGraph — Making the AI Actually Do Things

This is probably one of the parts I'm most interested in.

I don't want the LLM to directly interact with my database and randomly decide what to do.

Instead, I'm giving the agent specific tools.

For example:

search_product()
check_inventory()
get_dealer_price()
get_scheme()
get_customer_history()
create_draft_order()

So a conversation could look something like:

Dealer:
"I need 50 units of Product X."

AI:
"Let me check the availability."

    ↓

check_inventory("Product X")

    ↓

Inventory:
72 units available

    ↓

AI:
"We have 72 units available. Would you like me to create the order for 50?"

    ↓

Dealer:
"Yes."

    ↓

create_draft_order(...)

This is where LangGraph becomes useful.

Instead of having one massive prompt trying to handle the entire business workflow, the agent can move through different states and use specific tools.

Conceptually:

START
↓
Understand Request
↓
Identify Product
↓
Check Inventory
↓
Check Dealer Pricing
↓
Apply Scheme
↓
Confirm Order
↓
Create Draft Order
↓
END

This also makes debugging much easier.

If something goes wrong, I can ask:

Which step failed?

Instead of:

Why did the AI randomly do that?

1. PostgreSQL — Where the Real Data Lives

One thing I definitely don't want is for the AI to become the source of truth.

The LLM can understand things.

It can reason.

It can communicate.

But it shouldn't be the database.

The system needs structured data for things like:

• Dealers
• Dealer contacts
• Dealer addresses
• Products
• Product variants
• SKUs
• Inventory
• Orders
• Pricing
• Schemes
• Customer preferences

For example:

Dealer
├── Contacts
├── Addresses
├── Credit Limit
├── Language Preference
└── Preferred Warehouse

Product
├── Product Variant
├── SKU
├── Pricing
└── Inventory

I'm using PostgreSQL for this persistent data.

I'm also using UUIDs for primary keys and keeping things like timestamps, foreign keys, indexes, and soft-delete support in the domain design.

I'm still refining the schema, but getting this foundation right is important because adding AI on top of a messy data model isn't going to magically fix it.

1. Redis — The Fast Stuff

Not everything needs to be stored permanently in PostgreSQL.

A voice conversation can have a lot of temporary state.

For example:

• Current session
• Conversation state
• Temporary context
• Caching
• Rate limits

That's where Redis comes in.

The mental model I'm using is basically:

PostgreSQL
= Persistent business data

Redis
= Fast temporary state + caching

1. Inventory Is Where Things Get Real

Let's say a dealer says:

"I need 100 units."

The AI can't just say:

"Sure, I've placed the order."

It needs to check what's actually available.

Something like:

Dealer Request
↓
Identify SKU
↓
Inventory Service
↓
Available Quantity
↓
AI Response

If the system only has 60 units, the AI should say:

"We currently have 60 units available. Would you like me to create the order for 60?"

This is a principle I'm trying to stick to throughout the project:

The AI should reason about business data, not invent business data.

1. Dealer-Specific Pricing

This is another place where a normal chatbot approach isn't enough.

In B2B, everyone doesn't necessarily get the same price.

Different dealers might have:

• Different price lists
• Different discounts
• Different schemes
• Different credit limits
• Different warehouses
• Different purchasing histories

So the flow could look like:

Base Product Price
↓
Dealer-specific Price
↓
Applicable Scheme
↓
Discount
↓
Final Price

But here's an important architectural decision:

The LLM shouldn't calculate the final business-critical price itself.

The backend should do that.

The AI can explain the result to the dealer.

The actual calculation should come from deterministic business logic.

1. Customer Memory

This is probably one of the coolest parts of the idea.

A useful sales agent shouldn't feel like it has amnesia after every phone call.

Suppose a dealer usually orders a particular product or prefers a particular warehouse.

The system could remember useful information such as:

• Preferred warehouse
• Preferred products
• Typical order quantity
• Preferred language
• Previous orders
• Communication preferences

Then a future conversation could be much smoother.

Instead of asking:

"Which warehouse do you want?"

every single time, the system could already know the dealer's preferred warehouse and simply confirm it when needed.

But there's an important distinction here.

Not everything the dealer says should automatically become permanent memory.

Memory needs rules.

Some information is temporary conversation context.

Some information is actual customer data.

And some information probably shouldn't be stored at all.

That's something I want to handle carefully as the project develops.

1. ERP Integration

Eventually, the AI needs to connect with the systems that the business already uses.

That's where ERP integration comes in.

The architecture I'm aiming for is:

AI Agent
↓
Backend
↓
Business Logic
↓
ERP APIs
↓
Orders / Inventory / Customers

I don't want the AI agent directly modifying ERP data.

Instead:

AI
↓
Tool
↓
Backend Validation
↓
ERP

That gives us a much safer boundary between the unpredictable nature of AI and the deterministic nature of enterprise systems.

1. Why I'm NOT Giving the LLM Direct Database Access

This is probably one of the biggest things I've learned while designing this.

It can be tempting to just give an LLM database access and say:

"Do whatever you need."

But for a system dealing with real orders, pricing and customer information, that's a very bad idea.

I'd rather have:

❌ LLM → Database

✅ LLM → Tool → Backend → Database

For example:

LLM
↓
get_inventory("SKU123")
↓
Backend validates request
↓
Database query
↓
Structured result
↓
LLM

Now I have a proper place for:

• Authentication
• Authorization
• Validation
• Logging
• Security
• Error handling
• Auditing

And most importantly, I know exactly what the AI is allowed to do.

The Tech Stack

Frontend: Next.js

Styling: Tailwind CSS + shadcn/ui

Backend: FastAPI

Database: PostgreSQL

Cache / State: Redis

ORM: SQLAlchemy

Migrations: Alembic

Voice: Twilio

Real-time AI: OpenAI Realtime API

Agent Orchestration: LangGraph

Vector Search: pgvector

Version Control: Git + GitHub

Monitoring: Sentry + OpenTelemetry

I'm trying to avoid choosing technologies just because they're popular.

I want every piece of the stack to have a reason for being there.

What I'm Still Figuring Out

The project is still a work in progress, so there are a lot of things I'm actively figuring out.

1. Voice latency

A voice agent has to feel like a conversation.

Even a technically correct answer feels bad if it takes too long.

1. Hallucinations

The agent absolutely cannot randomly invent:

• Product availability
• Prices
• Discounts
• Order status
• Credit information

Those things need to come from actual systems.

1. Conversation recovery

What happens when the dealer says:

"No, not that one. The other blue one."

The agent needs enough context to understand what they're referring to.

1. Tool failures

What if the inventory service is down?

The AI shouldn't pretend that everything worked.

It needs to understand that the tool failed and communicate that properly.

1. Security

Once you're dealing with actual business transactions, security becomes a major part of the architecture.

Things like:

• Authentication
• Authorization
• PII protection
• Call verification
• Audit logs
• Tool permissions

can't just be an afterthought.

The Biggest Thing I've Learned

The biggest lesson from this project so far is:

Building an AI application isn't the same as putting an LLM inside an application.

The LLM is only one part of the system.

A useful AI product needs:

AI
+
Business Logic
+
Data
+
Tools
+
State
+
Security
+
Observability
+
Reliable Infrastructure

The model provides the intelligence.

But the architecture around it provides the reliability.

And I think that's an important distinction, especially when moving from AI demos to actual products.

What's Next?

Right now, I'm focusing on building the foundation properly before trying to make everything "smart."

The next things on my list are:

• Building the core backend APIs
• Implementing the voice pipeline
• Connecting the agent to business tools
• Building the order workflow
• Adding customer memory
• Connecting inventory
• Designing ERP synchronization
• Adding observability
• Testing real-world conversations

The end goal is pretty simple:

A dealer should be able to pick up a phone and complete a business transaction through a natural conversation.

Something like:

Call
↓
Talk
↓
Confirm
↓
Order

That's the experience I'm trying to build.

Final Thoughts

This project has also changed the way I think about software engineering.

Earlier, I mostly thought about applications as:

Frontend
+
Backend
+
Database

Now I'm asking a lot more questions:

What should the AI be allowed to decide?

What should the backend decide?

Where does the actual source of truth live?

What happens when the AI is wrong?

What happens when a tool fails?

How do we recover from a misunderstood request?

How do we make the whole thing observable?

And honestly, I'm still figuring out many of these answers.

That's probably my favorite part of building this.

I'm not trying to pretend I have the perfect architecture figured out.

I'm building it, breaking things, learning, and improving it as I go.

Build → Break → Learn → Improve.

If you're also building AI agents, voice applications, or AI-powered SaaS products, I'd love to hear what you're working on and what problems you've run into.

Let's learn from each other. 🚀