RAG Architecture Diagram: the Boxes and What Each Arrow Costs

Short answer: A RAG architecture diagram is five boxes and the arrows that join them - an index you build, a retriever that turns a query into candidates, a reranker that orders them, a context builder that packs what the model may read, and a loop controller that decides whether any of it runs again. Every arrow carries a payload, and every payload is paid for twice: once when it is moved, once when it is read.

Key takeaways

• Five boxes, six arrows, one loop. The loops are the difference between a retrieval feature and a bill with a cron schedule; the arrows are where the engineering actually sits.
• The index is the ceiling. A fact that was chunked away, split across two rows or dropped by a missing payload cannot be recovered by a reranker or a better prompt.
• The context builder is the only box that shrinks the bill. It is also the only one whose effect you can measure in the same request, and it reports its own ratio.
• Identity is not in the picture, and it has to be. No retrieval interface asks who is calling, so the caller's id has to travel from the session into the query filter.
• The loop controller is where cost escapes. A second iteration pays for every box above it again, which is why the cap belongs in the controller and not in a comment.
• Do this next: open your own stack, write down what crosses each of the six arrows, and price the widest one - the arrow into the prompt. That number is your monthly retrieval bill before any tuning.

The five boxes and the cost of every arrow between them

Draw the picture before choosing a vendor. A retrieval stack that survives production is five boxes joined by arrows, and each arrow carries a payload that someone pays for. The index holds what the retriever is allowed to find; the retriever turns one query into candidates; the reranker reorders those candidates; the context builder decides what the model finally reads; the loop controller decides whether the whole thing runs again. the architecture this diagram draws walks the same stack from the implementation side, with the code each box runs and the reason it is written that way. This page is the diagram itself: the boxes, the arrows, and the price attached to each arrow.

Read the table as a budget rather than an org chart. The left column is what you build once; the right column is what runs per request. The two arrows that grow with data volume - candidate vectors and scored candidates - are the ones to watch, because they grow even when traffic does not. And the last row is the one most diagrams leave out, which is why it is the one that shows up in incident reports.

The index box: what the retriever is allowed to find

The index is the only box with a build step, and it fixes the ceiling for everything downstream. If a fact was chunked away, split across two rows, or dropped because its payload was incomplete, no reranker and no prompt rewrite will bring it back - the retriever can only return what the index already holds.

Three things travel together in every row: the vector, the text the model will eventually read, and the metadata a filter matches on. Two of them are cheap to change later. Swapping a metadata field means backfilling the existing rows; swapping the embedding model means re-embedding the whole corpus, because vectors from two models do not share a coordinate system. Chunking sits in the expensive column too, and it is the decision people make last: a 512-token window that cuts a table in half produces two vectors matching neither the table nor the question, and the retriever will confidently return both.

A different index shape is legitimate. Zep's temporal knowledge graph stores facts with validity windows rather than passages, which answers "what was true last Tuesday" - a question a passage index cannot express - at the price of an extraction pipeline that has to run before anything is searchable. The diagram does not change; only what the first box holds changes.

The retriever box: a filter, a top-k and a score you did not calibrate

The retriever is the shortest code in the stack and the most common place to lose an afternoon. Approximate nearest-neighbour search returns the k closest vectors, and that list is a ranking, not a verdict: a similarity of 0.82 says nothing about whether the passage answers the question. Two decisions belong to this box, and both of them are about the arrow it produces.

First, filters belong inside the store rather than in application code afterwards. Post-filtering a k of 20 can leave two usable passages, and the effective k then changes with the data rather than with your configuration. Second, k is not a quality dial on its own: every extra candidate is one more passage the reranker has to score and one more chance for an irrelevant passage to reach the prompt. Measure recall at k of 5, 10 and 20 once, on your own questions, and you will know which of the two failure modes you have - starving the reranker, or diluting it.

The reranker: the box that buys precision with latency

A reranker is the easiest box to delete and the easiest box to under-budget. A cross-encoder reads the query and one passage together and returns a score for that pair, which means its work is linear in k: fifty candidates are fifty forward passes inside the request path. That is the trade in one sentence - precision at the top of the list, paid for in milliseconds that the user waits through.

Two conditions make the box worth its latency. The retriever must return more candidates than the prompt can hold, so that there is something to reorder; and the order must matter, which is true when the model reads the top few passages closely and ignores the rest. When the retriever already returns three passages that all belong in the answer, the reranker is decoration. When it returns thirty ranked by vector distance alone, the reranker is the difference between an answer and a plausible-sounding miss.

The context builder: where the protocol ends and the budget starts

The context builder is where the diagram stops being free. Everything above it produces candidates; this box decides how many of them the model may read, and it is the only box in the stack that can shrink the bill without losing the answer. The excerpt below is the arithmetic: the target token count is either given directly or derived from a compression rate, and the rate is then handed to the segmentation step, which splits the context into segments that carry their own settings.

# backend/smartgate/modules/context_gate/algorithm.py — source lines 366–373 (PromptCompressor.structured_compress_prompt)
        Returns:
            dict: A dictionary containing:
                - "compressed_prompt" (str): The resulting compressed prompt.
                - "origin_tokens" (int): The original number of tokens in the input.
                - "compressed_tokens" (int): The number of tokens in the compressed output.
                - "ratio" (str): The compression ratio achieved, calculated as the original token number divided by the token number after compression.
                - "rate" (str): The compression rate achieved, in a human-readable format.
                - "saving" (str): Estimated savings in GPT-4 token usage.

# backend/smartgate/modules/context_gate/algorithm.py — source lines 387–405 (PromptCompressor.structured_compress_prompt)
        if target_token == -1:
            target_token = (
                (
                    instruction_tokens_length
                    + question_tokens_length
                    + sum(context_tokens_length)
                )
                * rate
                - instruction_tokens_length
                - (question_tokens_length if concate_question else 0)
            )
        else:
            rate = target_token / sum(context_tokens_length)
        (
            context,
            context_segs,
            context_segs_rate,
            context_segs_compress,
        ) = self.segment_structured_context(context, rate)

Two details in that code are worth copying into your own diagram. The instruction and the question are counted before the rate is applied and subtracted afterwards, so the budget covers the context rather than the whole request - a mistake here quietly under-compresses by exactly the size of the instruction. And the return value is not the compressed text alone: it carries the original token count, the compressed token count, the achieved ratio and an estimated saving, which makes the box measurable in the same request that pays for it. The same five boxes also carry a set of choices - which one to swap, what each swap costs in latency, and which failure each choice invites - and that argument is the whole job of the decisions behind these boxes. If you want the wider catalogue of tricks before changing this box, token optimization techniques covers the ones that are not compression.

The context architecture: one rate per segment, not one rate per request

One compression rate for a whole request is a blunt instrument, and the code in most stacks assumes it. The alternative is visible in the next excerpt: the context arrives as segments, each tagged with its own rate and a compress flag, and the builder normalises, defaults and validates them before anything is compressed.

# backend/smartgate/modules/context_gate/algorithm.py — source lines 2100–2112 (PromptCompressor.segment_structured_context)
        for text in context:
            if not text.startswith("<llmlingua"):
                text = "<llmlingua>" + text
            if not text.endswith("</llmlingua>"):
                text = text + "</llmlingua>"

            # Regular expression to match <llmlingua, rate=x, compress=y>content</llmlingua>, allowing rate and compress in any order
            pattern = r"<llmlingua\s*(?:,\s*rate\s*=\s*([\d\.]+))?\s*(?:,\s*compress\s*=\s*(True|False))?\s*(?:,\s*rate\s*=\s*([\d\.]+))?\s*(?:,\s*compress\s*=\s*(True|False))?\s*>([^<]+)</llmlingua>"
            matches = re.findall(pattern, text)

            # Extracting segment contents
            segments = [match[4] for match in matches]

# backend/smartgate/modules/context_gate/algorithm.py — source lines 2127–2139 (PromptCompressor.segment_structured_context)
            segs_compress = [
                compress if compress is not None else True for compress in segs_compress
            ]
            segs_rate = [
                rate if rate else (global_rate if compress else 1.0)
                for rate, compress in zip(segs_rate, segs_compress)
            ]
            assert (
                len(segments) == len(segs_rate) == len(segs_compress)
            ), "The number of segments, rates, and compress flags should be the same."
            assert all(
                seg_rate <= 1.0 for seg_rate in segs_rate
            ), "Error: 'rate' must not exceed 1.0. The value of 'rate' indicates compression rate and must be within the range [0, 1]."

The tags are an interface between boxes, and that is the architectural point rather than a detail. Whoever produces the context - the ingest step, the retriever, the loop controller - can mark a passage as untouchable, and the marking travels with the text instead of living in a configuration file that nobody re-reads. A passage holding a table, a worked example or a citation can be preserved whole while the surrounding prose is compressed hard. The defaults are equally deliberate: a segment with no rate inherits the global rate when compression is allowed and a rate of 1.0 when it is not, so the two settings can never contradict each other, and the asserts refuse a rate above 1.0 or a segment whose counts disagree. A stack that compresses uniformly is not wrong, it is simply leaving its easiest saving on the table - context window management techniques covers the surrounding budget questions.

The scoring pass: a second model call, no protocol involved

Compression is usually described as a text operation, and the second model to run in your request path is usually left out of the diagram. It is there in the excerpt: the candidate chunks become a dataset, the dataset is batched through a small trained model with gradients switched off, the logits go through a softmax, and the per-token probabilities are merged back up into per-word probabilities.

# backend/smartgate/modules/context_gate/algorithm.py — source lines 2191–2200 (PromptCompressor.__get_context_prob)
        chunk_probs = []
        chunk_words = []
        with torch.no_grad():
            for batch in dataloader:
                ids = batch["ids"].to(self.device, dtype=torch.long)
                mask = batch["mask"].to(self.device, dtype=torch.long) == 1

                outputs = self.model(input_ids=ids, attention_mask=mask)
                loss, logits = outputs.loss, outputs.logits
                probs = F.softmax(logits, dim=-1)

# backend/smartgate/modules/context_gate/algorithm.py — source lines 2215–2228 (PromptCompressor.__get_context_prob)
                    (
                        words,
                        valid_token_probs,
                        valid_token_probs_no_force,
                    ) = self.__merge_token_to_word(
                        tokens,
                        token_probs,
                        force_tokens=force_tokens,
                        token_map=token_map,
                        force_reserve_digit=force_reserve_digit,
                    )
                    word_probs_no_force = self.__token_prob_to_word_prob(
                        valid_token_probs_no_force, convert_mode=token_to_word
                    )

Read the cost of that box before adopting it. It is a second model, inside the request path, whose latency grows with the number of tokens you feed it - and whose output is only a ranking of tokens by how predictable they are, which the compressor then uses to decide what to drop. Nothing about it is a protocol feature: it runs in your process against your own weights, so it is invisible to the caller and to any gateway unless you log it. The configuration in the first excerpt is what keeps it honest: the token-level filter can be switched off entirely, and the context-level and sentence-level filters can be used alone, which is the right answer when the second model's latency is larger than the saving it produces.

Caller identity: the box no specification asks for

Nothing in the diagram so far knows who is asking. Retrieval interfaces take a query and a filter; they do not take a principal, and no protocol specification turns a session into an authorization decision. So identity has to be carried into the stack by whichever box already knows it, and the excerpt below is the shape it usually has: a session callback that copies the user id out of the token subject and keeps the role alongside it.

# auth.ts — source lines 39–48 (session)
async session({ token, session }) {
      if (session.user) {
        if (token.sub) session.user.id = token.sub;
        (session.user as { role?: UserRole }).role = (token.role as UserRole) || UserRole.USER;
        session.user.name = token.name ?? null;
        session.user.email = token.email ?? "";
        session.user.image = token.picture ?? null;
      }
      return session;
    }

That copied id is the only thing standing between a shared index and a per-tenant leak, and it is a filter value rather than a security boundary. Two consequences follow. The id has to reach the retriever, which means the filter is built at the edge of the stack rather than deep inside the vector code, where it would have to be threaded through every call site. And it has to be recorded with the request, because a retrieval that cannot name its principal cannot be audited afterwards. Agent memory has the same shape - a store keyed by user, agent and run - and agent memory architecture works through the write path when the store fills up.

The last arrow: one status code behind a streamable HTTP endpoint

An architecture diagram usually stops at the model's answer. The arrow that matters operationally is the one leaving the application: what the caller receives when one of the boxes fails. The excerpt is the mapping - a result object becomes a response envelope, and if the envelope says the call failed, the caller gets an HTTP error with the failure detail attached instead of a partial answer.

# backend/smartgate/core/models.py — source lines 38–46 (smartgate_http_response_from_result)
def smartgate_http_response_from_result(result, *, status_code: int = 422):
    """Map ToolResult → API envelope; failed tools become HTTP errors (REST clients)."""
    from fastapi import HTTPException

    response = smartgate_response_from_result(result)
    if not response.success:
        detail = response.error or {"code": "ERROR", "message": "request failed"}
        raise HTTPException(status_code=status_code, detail=detail)
    return response

One status per failure family beats one status for everything, because the transport layer is where retries, budgets and alerts are decided. A caller that can tell a validation failure from a capacity failure from an upstream timeout retries the right things and pages the right person; a caller that receives a 200 with an empty body retries everything and learns nothing. The same envelope discipline is what lets a gateway sit in front of the stack at all, which is the argument in what a gateway adds: limits, keys and audit rows only make sense if a failed call is visible as a failure.

One architecture rule: every box returns a shape the next box can read

Every arrow in the diagram is a contract, and the cheapest way to break one is to return nothing. The excerpt is the smallest version of that bug and its fix, from a documentation source config: a node with no children is given a single space, so the renderer below it never has to handle an empty list.

# source.config.ts — source lines 49–53 (onVisitLine)
onVisitLine(node: { children: unknown[] }) {
            if (node.children.length === 0) {
              node.children = [{ type: "text", value: " " }];
            }
          }

That is a guard against an absent shape, not against empty content. An empty result is fine; a missing field is not. In a retrieval stack the same rule applies three times over: a retriever that finds nothing must return an empty list with the same fields rather than a null; a context builder that drops every passage must still emit a prompt, even a short one that says so; and a loop controller that stops early must still return the shape the caller expects. Teams that skip this spend their first week of production debugging missing keys instead of missing answers, and an agentic stack multiplies the surface, because every extra box and every extra iteration is another chance to hand the next box a hole - what agentic RAG changes is the version of this diagram where the loop owns more of the boxes.

The loop controller: the only box that can run away with the bill

Everything so far runs once. The loop controller is the box that decides to run it all again, and it is the only component in the diagram whose failure mode is a bill rather than a wrong answer. A second iteration means a rewritten query, a fresh retrieval, another rerank and another packed prompt; the marginal cost is the whole stack, and the marginal benefit is the one passage the first pass missed.

Three limits belong to this box rather than to the caller, because the caller cannot see the loop. A cap on iterations, because the third pass almost never pays. A cap on tokens per request, because the loop's cost is dominated by what each pass carries into the prompt. And a budget check before the second pass rather than after it, because a check that runs at the end of the request reports the overrun instead of preventing it. Measuring this box is a cost question with a known shape - what a retrieval loop costs takes the same stack apart in money - and the loop is also the only place where a retrieval stack becomes an agentic one, which is the line RAG versus agentic RAG draws in detail.

Servers, tools and why a server list is not a retriever

One confusion costs teams an integration. A tool server exposes capabilities a model can call with arguments, and a directory of such servers is a catalogue of who offers what. A retriever answers a different question: given a query, which passages in my corpus are worth reading. Both end up inside the context window, and only one of them has a reranker, a similarity threshold and a k to tune.

The practical test is the arrow test. If the answer arrives because the model asked for it with a specific argument, it is a tool call, and the tool surface a host exposes describes how those arguments and results are shaped. If the answer arrives because the system searched for it before the model was invoked, it is retrieval. A server list can be a source of documents - fetch a page with a tool, hand it to the ingest step, index it - but the list itself is not an index, and no reranker will ever see it. The academic version of that boundary, and the design space on both sides of it, is the subject of the agentic RAG survey.

Which of these boxes you own at all — and which belong to a runtime that decides for itself when to
retrieve — is the distinction drawn in rag vs agentic ai, which is
the page to read when the diagram and the deployment disagree about who is in control.

How SmartGate compares

The interesting comparison is not which box is better but who owns which box. The five-box diagram is the same for everyone; the difference is how much of it you have to build, monitor and pay for.

The reason to hand over the middle of the stack is that the middle is where the money leaks: compression, deduplication and the loop's caps are the boxes that decide the size of the prompt. Free tier covers 2 million tokens a month with all seven tools and no card; Pro starts at 18 dollars a month, Teams at 55, with the same per-key limits on every plan.

How to get started

1. Write down your six arrows. For each one, note what crosses it and whether it scales with data volume or with traffic. The arrow into the prompt is the one to price first.
2. Find the arrow with no owner. The layer around the boxes - identity, limits, logs - is the one that appears in no diagram and in every incident.
3. Switch on the context builder before the reranker. Compression changes the size of every subsequent call; a reranker changes the order of one list.
4. Pick k from a measurement, not from a habit. Recall at 5, 10 and 20 on your own questions takes an afternoon and settles the reranker argument at the same time.
5. Cap the loop before you ship it. Iterations, tokens per request and a budget check that runs before the second pass.
6. Log the principal with the retrieval. A request that cannot name its caller cannot be audited, quoted or billed to anyone.

Start on the free tier - 2 million tokens a month and all seven tools - from start free; the per-plan limits sit on the pricing page, the endpoint shape is in the product docs, and contract traffic begins at the contact form.

FAQ

What are the five boxes of a RAG architecture?
An index, a retriever, a reranker, a context builder and a loop controller. The index holds chunks, vectors and filter metadata; the retriever turns a query into candidates; the reranker reorders them; the context builder packs what the model reads; the loop controller decides whether the pass runs again. Identity, limits and logs sit around the boxes rather than inside them.

Is a reranker always worth the latency?
No. It is worth it when the retriever returns more candidates than the prompt can hold and the order of those candidates changes the answer. A cross-encoder scores one query and one passage at a time, so its cost grows with the number of candidates. If the retriever already returns three passages that all belong in the answer, the reranker is decoration.

Where does a retrieval stack usually break first?
At the arrow into the prompt. The index is the ceiling on recall, but the prompt is the ceiling on the bill, and a stack that retrieves well and packs badly pays for tokens the model does not need. The second most common break is a box that returns an empty result with the wrong shape, which surfaces as a missing key rather than a missing answer.

Can I skip the index and call a search API instead?
Yes, and for a small corpus it is the right first move: a hosted search API becomes your index and retriever in one, and you keep the context builder and the loop. What you give up is control of chunking, filter semantics and score calibration, which are exactly the three knobs you need when recall stops improving.

How do I know the context builder helped?
Measure the token count before and after the same request. A builder worth keeping reports the original token count, the compressed token count and the achieved ratio, so the saving is an observation rather than a belief. Compare the trimmed prompt with the untrimmed one on the same questions: if the answers are identical and the token count halved, the box is doing its job.

Does the diagram change for an agentic stack?
The boxes stay and the loop grows. An agentic system adds iterations, so the loop controller owns more of the cost and needs caps of its own, and every extra box is another shape that has to reach the next box intact. The arrows still carry the same payloads; there are simply more passes over them.

Limitations and what this does not do

• This is one stack, not a vendor map. The excerpts come from one production implementation of the boxes. Another stack will have different class names and a different order of operations, and the arrows will still be the six in the table.
• Two of the eight sections carry no code. The index-shape section and the servers-and-tools section matched no unique symbol in the codebase, so they are written from the published paper and the vendors' own material, with no quoted implementation. Where a section shows no fence, that is the reason.
• The cost figures in the tables are structural, not measured. "Linear in k" and "a full pass" describe how each arrow scales; they are not benchmarks from a particular machine or corpus.
• The identity section is about carrying a principal, not about authentication. It shows where the caller's id comes from and why it has to reach the filter. The authorization decisions behind that id are a separate subject.
• The diagram assumes text. Images, audio and structured records enter the same pipeline through the ingest step, and every box after it behaves differently for them.

Sources

• Zep, a temporal knowledge graph architecture for agent memory: https://arxiv.org/abs/2501.13956
• Cross-encoder reranking, sentence-transformers documentation: https://www.sbert.net/examples/applications/cross-encoder/README.html
• Language modeling is compression (the compression rate used by the context builder): https://arxiv.org/abs/2309.10668
• LLMLingua, prompt compression for accelerated inference: https://arxiv.org/abs/2310.05736
• Qdrant, filtered vector search and top-k semantics: https://qdrant.tech/documentation/concepts/search/
• Model Context Protocol, the tool and resource surfaces a host exposes: https://modelcontextprotocol.io/

Method note

The code in this article is not transcribed. Each block was cut directly out of the slice body returned by the SmartGate slice API, then re-asserted byte for byte as a substring of that body before publication; the first line inside every fence records the file and the exact source lines. Symbols were pinned with whole-name containment (rule A level 2) and confirmed by the service's slot-proof endpoint - 6 of 8 planned sections pinned, no abstentions, and the two sections that matched no unique symbol are written from published sources with no code, as the abstain rule requires. Some blocks show two line windows rather than one: the gap between them is the part of the function that does not serve this page's diagram, so it is not quoted. The excerpts are one implementation's, chosen because the boxes in this order exist there in code rather than in prose.

Slice provenance

Every fenced block above was cut from the slice body and re-asserted against it byte for byte before
publication. 6 of 8 sections pinned, 0 abstentions, 2 misses.