Building an AI Customer Support SaaS with Django, RAG and Self-Hosted LLMs
Over the past several months, I’ve been building AI-Autofy, an AI customer-support SaaS designed to let businesses train an assistant on their own website, documents, FAQs and business data.
At first glance, building an AI chatbot sounds straightforward:
Send a prompt to an LLM.
Display the response.
Add a chat widget.
In practice, once you need reliable business-specific answers, tenant isolation, live data, product information, images, analytics and predictable inference costs, the architecture becomes considerably more interesting.
This post covers some of the main lessons I learned while building it.
The basic architecture
The web application is built with Python and Django.
Django handles things such as:
Customer accounts
Subscriptions
AI configuration
Knowledge-base management
Chat history
Analytics
Widget configuration
Tenant separation
Integrations
The AI inference layer is separated from the main Django application.
This means the web application does not need to run the language model itself.
Instead, requests are sent to an AI service responsible for generating responses.
Why separate the AI service?
Running an LLM inside the main web application creates several problems.
Inference workloads have very different requirements from normal web requests.
A typical Django request might take milliseconds, while an AI response may involve:
Retrieval
Prompt construction
GPU inference
Streaming tokens
Tool calls
Live-data lookups
Separating these workloads allows the web application and AI infrastructure to scale independently.
It also makes it possible to change the model without redesigning the SaaS application.
Retrieval-Augmented Generation
A customer-support assistant should not rely entirely on the model’s general knowledge.
A business wants the AI to answer questions using its own information.
For example:
What is your refund policy?
or:
Do you provide support outside Ireland?
The relevant information might exist on the company website or inside a PDF.
I therefore use a retrieval pipeline.
Conceptually:
Customer question
|
v
Create embedding
|
v
Search business knowledge
|
v
Retrieve relevant documents
|
v
Build LLM prompt
|
v
Generate grounded answer
The important part is tenant isolation.
A document belonging to Company A must never appear in a response generated for Company B.
Every retrieval request therefore needs to remain scoped to the current tenant.
A vector database is only part of the solution
I use a vector database for semantic retrieval, but retrieval quality depends on much more than simply storing embeddings.
Things that matter include:
Chunk size
Metadata
Tenant filtering
Similarity thresholds
Query rewriting
Number of retrieved chunks
Prompt construction
Retrieving too little context can produce incomplete answers.
Retrieving too much can fill the context window with irrelevant information.
I found that relevance filtering is one of the most important parts of the system.
Static knowledge versus live data
A vector database works well for relatively static information.
But consider a question such as:
What products are currently available?
That information may change constantly.
Embedding yesterday’s product catalogue is not necessarily the right solution.
I therefore treat knowledge data and live data differently.
Knowledge data includes things such as:
Website content
FAQs
Documentation
Policies
Live data can include:
Products
Prices
Availability
Business-system information
The interesting problem is deciding when live data should be queried.
You do not want a product catalogue added to every prompt simply because it exists.
The user’s question needs to be relevant first.
Relevance gating
This turned out to be an important lesson.
Imagine a customer asks:
What are your opening hours?
If the application automatically injects product data into every request, the LLM may start mentioning products even though the question has nothing to do with them.
The same applies to images.
The solution is to introduce relevance checks before enriching the prompt.
Conceptually:
if is_product_question(message):
context += get_product_data()
The real implementation can be more sophisticated, but the principle is simple:
Give the model additional information only when it is relevant.
This improves both response quality and token efficiency.
Dynamic images have the same problem
AI responses can become much more useful when they contain relevant images.
For example, if somebody asks:
Can you show me the blue version?
an image may be very helpful.
But displaying an image because a generic keyword happened to match makes the assistant feel unreliable.
So image selection also needs relevance filtering.
A useful AI interface is not about showing everything available.
It is about showing the right information at the right moment.
Self-hosting the language model
One of the biggest architectural decisions was how to handle inference.
Using hosted AI APIs is extremely convenient, particularly during development.
However, predictable SaaS pricing becomes harder when every customer interaction has a variable external API cost.
I therefore experimented with self-hosted models running on GPU infrastructure.
The architecture is roughly:
Website Widget
|
v
Django
|
v
AI Service / Agent
|
+---- Vector Database
|
+---- Live Data
|
+---- LLM Inference
This gives more control over:
Model choice
Token limits
Capacity
Cost per message
Scaling
Data flow
There are trade-offs, of course.
Running inference infrastructure means dealing with GPU availability, model loading, monitoring and capacity planning.
Streaming responses
For chat applications, perceived latency matters almost as much as total generation time.
Waiting several seconds and then receiving an entire response feels much slower than seeing the answer appear incrementally.
Streaming therefore makes a significant difference to the user experience.
The flow becomes:
Browser
|
| question
v
Django
|
v
AI service
|
| token stream
v
Django
|
| streamed response
v
Browser
Even when total generation time remains similar, the application feels considerably more responsive.
Human escalation still matters
An AI support system should not pretend it can solve everything.
There are situations where a human should take over.
Examples include:
Complaints
Sensitive account issues
Missing information
Complex requests
Situations requiring human judgement
One of the design principles I have adopted is:
The AI should know when it does not have enough information.
That is more useful than confidently inventing an answer.
Multitenancy changes everything
Building an AI demo for one business is relatively easy.
Building a SaaS where hundreds of businesses can independently configure their assistants is different.
Each tenant may have:
Different instructions
Different knowledge
Different products
Different widgets
Different usage limits
Different conversation histories
Every step of the pipeline must preserve tenant context.
That includes retrieval, live-data access, logging and analytics.
Cost becomes an architectural feature
When building a SaaS product, AI cost is not just an infrastructure concern.
It directly affects the business model.
If a customer pays €20 per month, the platform cannot consume €30 of inference infrastructure serving that customer.
This means decisions such as these become important:
Context length
Number of retrieved documents
Model size
GPU utilization
Message limits
Caching
Prompt size
Concurrency
AI efficiency becomes part of product engineering.
What I would do differently
If I were starting again, I would spend more time on relevance and retrieval quality earlier.
It is tempting to focus on model size.
But for a customer-support system, a smaller model with excellent business context can often be more useful than a larger model receiving poor context.
I would prioritize:
Good retrieval
Strong tenant isolation
Relevance gating
Clear system instructions
Fast streaming
Reliable fallbacks
before spending too much time experimenting with larger models.
The result
These ideas eventually became part of AI-Autofy.
The platform lets businesses add their website, documents, FAQs and instructions, configure an AI assistant, test it and deploy it using a website widget.
I’ve also been adding capabilities for live business information, products, images, multilingual conversations, escalation and analytics.
You can see the project here:
I’m continuing to work on both the product and the infrastructure behind it.
For anyone else building AI SaaS products, I’d be interested to hear how you are approaching the same trade-off between model quality, inference cost and retrieval quality.
