A website may already contain the answers visitors need.
The real problem is often finding them.
So I built and deployed a first version of a RAG AI assistant for my own website, with one particular constraint:
The response model runs directly in the visitor's browser using WebGPU.
This was not meant to be another chatbot connected to an API.
I wanted to explore what happens when retrieval, local inference, source-based answers, conversation context, and a CMS are designed as one complete workflow.
How the architecture works
The system splits the work between the website server and the browser.
1. The CMS prepares the knowledge base
Selected website content is split into passages and indexed.
The CMS also lets me manage:
- which documents are available to the assistant
- model profiles
- search parameters
- assistant instructions
- activation and validation
2. The browser understands the question
The browser can use recent conversation context to reformulate follow-up questions.
An embedding model then converts the query into a vector.
3. The server performs vector search
The vector is sent to the website API.
The server searches the document index and returns the most relevant passages together with their sources.
4. WebGPU generates the answer locally
The response model runs on the visitor's GPU.
It receives:
- the question
- recent conversation context
- retrieved passages
- CMS instructions
The first result is not displayed immediately.
A second pass checks the draft against the retrieved sources, and the system can perform one corrective search if necessary.
5. The visitor gets an answer with sources
The final response contains links back to the original website content.
If the available information is not sufficient, the assistant is expected to say so instead of inventing an answer.
Why run the model in the browser?
The main experiment was to avoid calling a hosted generation API for every message.
This changes several things.
No per-response generation API call
Once the models are loaded, generation happens on the visitor's device.
That does not make the system free — hosting, downloads, development and maintenance still have costs — but there is no token bill for every generated response in this workflow.
More control over conversation data
The conversation is not sent to an external generation API to produce the answer.
Optional conversation memory is also stored locally in the browser.
The server is still involved in document retrieval, so this is not a fully offline system.
Local memory
I also added optional semantic memory using IndexedDB.
It can keep previous exchanges locally for a limited period and retrieve an older, related question when useful.
An important design decision was to keep three concepts separate:
- documents are the factual source
- recent conversation maintains context
- local memory helps recover previous intent
An old AI-generated answer never becomes a new factual source.
The limitations are just as important
Running AI models in the browser comes with real constraints.
The current response model requires roughly 2 GB of downloads, plus about 279 MB for the embedding model.
Browser support is another factor.
Having navigator.gpu available does not automatically mean that every model and inference engine will work correctly on that device.
Drivers, GPU capabilities, memory limits and browser implementation all matter.
Latency is also very different from a fast cloud API.
With the models already cached, one documented test produced a reviewed answer in:
- 36.11 seconds on a Samsung Galaxy S25 Ultra
- 13.06 seconds on a desktop with an NVIDIA Ampere GPU
These are measurements from my own test devices, not performance guarantees.
More complex mobile requests can take over a minute.
Why I still find this architecture interesting
The goal is not to replace every search box with an AI assistant.
The interesting use cases are situations where users have a real question and the website already has reliable information that can answer it.
For example:
- product catalogs
- technical documentation
- training websites
- tourism or accommodation
- business software
- real-estate search
A user can express a need naturally, while structured data and RAG keep the response connected to information that can actually be verified.
The model is only one part of the solution.
The real value comes from combining:
data + retrieval + business rules + UX + performance + validation
What I learned from this first version
Building this project reinforced something I increasingly see in applied AI work:
A better model does not automatically create a better product.
The difficult parts are often around the model:
- preparing reliable knowledge
- keeping retrieval consistent
- managing context
- handling model downloads and caching
- keeping the interface responsive
- validating answers
- testing on real devices
- deciding when the system should simply say "I don't know"
This first version is now running in production on my own website, and I am continuing to test lighter models, additional browsers, GPUs and possible business applications.
I wrote a more complete article with the architecture, limitations, benchmarks and potential use cases:
👉 RAG and WebGPU: When Your Website Becomes an AI Assistant
There is also a deeper technical case study covering the complete RAG pipeline, CMS integration, embeddings, local memory and model choices.
