SmartGateSmartGate

What Is RAG Architecture? The Decisions You Cannot Defer

RAG architecture is the set of decisions inside a retrieval pipeline rather than the pipeline itself - which components you can replace without rewriting the rest, what each replacement costs in latency, and which failure it makes more likely. The boxes rarely change; the arguments are about what you put in them, and every choice that buys quality pays for it in milliseconds the user waits.

Short answer: RAG architecture is the set of decisions inside a retrieval pipeline rather than the pipeline itself - which components you can replace without rewriting the rest, what each replacement costs in latency, and which failure it makes more likely. The boxes rarely change; the arguments are about what you put in them, and every choice that buys quality pays for it in milliseconds the user waits.

Key takeaways

  • Four decisions carry most of the consequence. What the index treats as truth, who is allowed to retrieve, how much retrieval runs before the model is called, and which protocol carries the call.
  • Swappability is a property of interfaces, not of vendors. A component is replaceable when the contract around it names the score scale, the filter language and the default depth rather than the store.
  • Quality and latency are the same dial. Deeper retrieval, a reranker, less compression and a second loop iteration all move answer quality and response time together.
  • Every choice invites a specific failure. Stale answers come from index drift, uneven answers from two read paths with different defaults, and quiet degradation from a malformed configuration list.
  • Configuration is where a swap becomes real. A component you can only replace by editing code is not swappable in practice, however clean its interface looks on a diagram.
  • Do this next: list the components of your own stack and mark each one with the three facts a caller needs - score scale, filter language, default depth. The unmarked ones are where the next migration will hurt.

What the decisions are, and what they are not

The useful reading of RAG architecture is a list of choices that have to be made deliberately, because the defaults are all reasonable and all wrong for somebody. Which component may be replaced without rewriting the others? What does that replacement cost in latency, and what does it buy in answer quality? Which failure does each option make more likely? Those three questions are the whole discipline, and they are asked per component rather than once for the pipeline.

Two framing notes before the decisions. The boxes themselves - index, retriever, reranker, context builder, loop controller - are fixed, and the architecture these decisions live in covers the implementation side with the code each one runs. And the decisions are not equally expensive to reverse: swapping a model or a reranker is an afternoon, while re-chunking a corpus and rebuilding an index is a project. Sort the list by how much it costs to undo, and the arguments get much shorter.

Decision one: the retrieval boundary is a security boundary

Retrieval is the one component in the stack that hands untrusted text to a model with the authority to act. Everything the retriever returns is treated as context by whatever reads it next, so the question "who may retrieve what" is not a permissions nicety - it decides how much damage one bad document can do.

Three consequences are worth deciding on purpose. Filters must be built from server-side identity rather than from a value the caller passes, or a caller who can spell another tenant's id can read their corpus. Retrieved content must be treated as data, never as instructions, which means the prompt has to separate the two explicitly and the tool layer must not let a passage trigger a call - prompt handling on the read path works through the injection cases this creates. And the index itself is a place secrets accumulate: a corpus built by scraping internal pages will happily store a key somebody pasted into a wiki, and nothing downstream will notice. The failure this decision invites is the worst one in the list, because it is silent and it looks like a correct answer.

One read leaves this boundary entirely: when the retrieval path is a search tool somebody else operates, the corpus and the ranking stop being yours to protect, and the trade changes shape - agentic search vs RAG is where that version is priced.

Decision two: the index and the store are two resources that drift

The first decision that shows up in code is whether the listing of your content is derived from the store or maintained beside it. The excerpt below is a maintenance path for a listing: it checks each item's key against the store in parallel and keeps only the items that still exist.

# lib/forgenova-kv-index.server.ts — source lines 22–43 (validateIndexItems)
async function validateIndexItems(
  kind: string,
  items: PseoKVIndexItem[],
): Promise<PseoKVIndexItem[]> {
  if (!items?.length) return [];

  const results = await Promise.allSettled(
    items.map(async (item) => {
      const exists = await checkKVKeyExists(pseoContentKey(kind, item.slug));
      return { item, exists };
    }),
  );

  return results
    .filter(
      (
        r,
      ): r is PromiseFulfilledResult<{ item: PseoKVIndexItem; exists: boolean }> =>
        r.status === "fulfilled" && r.value.exists,
    )
    .map((r) => r.value.item);
}

The parallel existence check is the interesting part, because it is a reconciliation rather than a write. A derived listing can always be rebuilt, so its worst failure is a delay; an authored listing is a resource that can point at nothing, and the failure surfaces as a link a user clicks. In a retrieval stack the same pair exists one layer down: the vector index and the chunk table. A document deleted upstream leaves its vectors behind unless something reconciles them, and a retriever that returns a passage nobody can open any more hands the model a quote with no source. Decide which of the two is allowed to be wrong, and give the other one a reconciliation job - here it is the listing that gets checked, and it costs one lookup per item instead of a full rebuild.

Decision three: what an agent2agent caller cannot see about your retriever

The moment another agent calls your retriever, its internal implementation stops being private and its contract starts being a decision. The excerpt is the shape that contract usually takes: a collection name, a query vector, a depth with a default, and an optional filter, all passed to the store underneath.

# backend/smartgate/core/resources.py — source lines 141–149 (search_vectors)
async def search_vectors(self, collection: str, query_vector, top_k: int = 10, filters=None):
        from qdrant_client.models import Filter as QFilter
        result = await self._client.search(
            collection_name=collection,
            query_vector=query_vector,
            limit=top_k,
            query_filter=filters,
        )
        return result

Four things cross that boundary and each one is worth a written decision. The depth default is the first: a caller that omits it gets whatever the code chose, and an agent that always passes its own value will disagree with the caller that does not. The filter language is the second, because a caller that learns your filter shape has learned your store. The score scale is the third and the one that breaks quietly - similarity scores are not comparable across embedding models, so a threshold a caller calibrated against the previous store is meaningless after a swap and will look like a tuning problem. And the response shape is the fourth: an empty result must be an empty list with the same fields, not a missing key, or every caller needs its own defensive branch. If you plan to swap the store, publish this contract as a version and treat a change in score scale as a breaking change.

Decision four: the framework is a dependency, not an architecture

Frameworks are a genuine shortcut for the two hardest boxes - the retriever and the reranker - and a poor place to put the decisions that carry budgets. The test is not which framework is best; it is which decisions you can still make after you have chosen one.

Three should stay outside the framework even when the framework offers them: the query filter, because it is where identity and tenancy live; the context budget, because it is the only place cost is set; and the loop's limits, because a framework that retries and a controller that iterates are the same bill. Components that fit comfortably inside are the boxes with clean inputs and outputs - an embedding model, a similarity function, a chunker. A practical rule: if a component is behind an interface you control, keep it; if reaching it means importing the framework's types into your own domain code, it has become the architecture. And a framework that speaks a protocol can also be fronted by a gateway, which is where limits and audit rows belong - where the gateway sits is the layer that survives the next framework change.

Decision five: MCP or A2A decides who owns the retrieval call

The protocol choice is really a choice about who triggers retrieval. If the corpus is exposed as a tool the model can call, the model decides when to search, with what query, and whether to search again; if retrieval is a fixed step before the call, the application decides and the model only reads. Those two shapes produce different architectures from the same boxes.

The first shape is the host's tool surface: retrieval becomes one capability among several, and the agent gains the flexibility to search only when it needs to. It also gains variance - the same question may produce one call or four - and the cost of that variance lands on the loop controller's cap. The second shape is deterministic and cheaper, and it answers badly exactly when the first retrieval missed. The delegation protocol matters for a different reason: when a task moves to another agent rather than to a tool, the receiving side owns its own retrieval, and the A2A protocol exists for that hand-off. Decide whether retrieval is a step or a capability, and write it down, because agents happily do both and pay twice.

Writing that decision down is easier once the two nouns are separated: rag vs agentic ai draws the line between a retrieval pattern and an agentic runtime, and says which decisions belong on which side of it.

Decision six: the memory store an OpenAI-compatible host never sees

Memory is the component most often swapped without the callers noticing, which is a feature until two callers disagree about defaults. The excerpt is a read path from a memory store: a filter that must name an entity, a depth with its own default, a validation pass, and a result shaped as a list.

# backend/smartgate/modules/memory/algorithm.py — source lines 2431–2445 (AsyncMemory.get_all)
async def get_all(
        self,
        *,
        filters: Optional[Dict[str, Any]] = None,
        top_k: int = 20,
        **kwargs,
    ):
        """
        List all memories.

        Args:
            filters (dict): Filter dict containing entity IDs and optional metadata filters.
                Must contain at least one of: user_id, agent_id, run_id.
                Example: filters={"user_id": "u1", "agent_id": "a1"}
            top_k (int, optional): The maximum number of memories to return. Defaults to 20.

# backend/smartgate/modules/memory/algorithm.py — source lines 2476–2492 (AsyncMemory.get_all)
        # Validate filters contains at least one entity ID
        if not any(key in effective_filters for key in ("user_id", "agent_id", "run_id")):
            raise ValueError(
                "filters must contain at least one of: user_id, agent_id, run_id. "
                "Example: filters={'user_id': 'u1'}"
            )

        limit = top_k

        keys, encoded_ids = process_telemetry_filters(effective_filters)
        capture_event(
            "mem0.get_all", self, {"limit": limit, "keys": keys, "encoded_ids": encoded_ids, "sync_type": "async"}
        )

        all_memories_result = await self._get_all_from_vector_store(effective_filters, limit)

        return {"results": all_memories_result}

Two decisions hide in those assertions. The first is that the filter is required: a read that names no user, agent or run is rejected rather than answered, which is the right default for a store that will hold several tenants' facts and the reason the refusal names the shape it wants. The second is that the depth has a default of its own, so a caller that does not care gets twenty memories and a caller that cares passes its own number. That is where uneven answers come from: a client that reads twenty memories and an agent that reads fifty see different worlds, and both are correct about the store. Put the default in one place, document it as part of the read contract, and let an OpenAI-compatible client keep calling the same endpoint while the store behind it changes. The memory write path has its own decisions, and the memory store's write path is where they are made.

Decision seven: OpenClaw config is where the swap actually happens

A component you can only replace by editing code is not swappable, however clean its interface looks. The mechanical half of that claim is smaller than it sounds: the excerpt below turns a delimited configuration value into a list, trimming entries and dropping the empty ones.

# lib/sitemap-config.ts — source lines 3–9 (parseConfigList)
function parseConfigList(val: string | null | undefined): string[] {
  if (!val) return [];
  return val
    .split(/[\n,]+/)
    .map((p) => p.trim())
    .filter(Boolean);
}

Three properties of that helper are worth demanding from every configuration your stack reads. It accepts both separators, so a value written on one line or several means the same thing to the reader. It drops empty entries instead of failing, which is what keeps a trailing separator from disabling a source. And it returns a list rather than an optional, so the caller never has to distinguish "not configured" from "configured as nothing". The decision this serves is that a source, a model or a reranker should be enabled or disabled by data - OpenClaw config, an environment value, a settings row - so that the swap happens in a deployment rather than in a branch. The failure to watch for is silent: a malformed list that parses to empty disables a source, the retriever narrows, and every answer still looks plausible.

Decision eight: the vscode view is not a second index

The last decision is about how many copies of your text exist. Ingest almost always converts something first - a page, a record, a PDF - and the converted text is what gets embedded and later shown to a human in whatever panel they read it in. The excerpt is that conversion: HTML in, markdown out, navigation and script elements stripped before the rest is post-processed.

# backend/smartgate/modules/fetch/html_converter.py — source lines 82–97 (convert)
def convert(self, html: str) -> str:
        from markdownify import markdownify as md

        # Step 1: markdownify 基础转换
        result = md(
            html,
            heading_style="ATX",
            bullets="-",
            code_language="",
            strip=["script", "style", "nav", "footer", "aside"],
        )

        # Step 2: GFM 后处理
        result = self.gfm.process(result)

        return result.strip()

The decision is whether the converted text is the single artefact both consumers read, or whether each surface converts for itself. Two conversions of the same page diverge quickly - one strips the nav, the other keeps it; one flattens tables, the other does not - and then the passage a user can see is not the passage the model scored. Treat the ingest output as the canonical text, embed that exact string, and let the vscode panel or any other editor render a view of it rather than a second conversion. The same rule covers chunking: the chunk boundaries belong to the artefact, so a highlight in the editor maps back to a real row in the index instead of to a paragraph that was assembled for display.

Latency against quality: four dials and what each one costs

Every decision above ends up as one of four dials, and the four move together: turning any of them towards quality costs latency and tokens, and turning them down saves both while inviting a different failure.

Dial Turned up What it buys What it costs The failure it invites
Retrieval depth (k) 20 candidates instead of 5 recall: the passage exists in the list a rerank pass per candidate, plus more text moved dilution: an irrelevant passage reaches the prompt
Reranking on, with a cross-encoder precision at the top of the list a scoring pass per candidate in the request path latency that grows with the dial above it
Compression a low ratio, per segment a smaller prompt at the same recall a second model call, and a risk of dropping detail an answer that loses the one number that mattered
Loop iterations two or three passes the passage the first query missed the whole stack again per pass a bill with no ceiling and a slower answer

Read the table column by column rather than row by row. The second column is the argument for spending; the fifth is the argument for a limit. A stack with no limits anywhere in the table is not tuned for quality, it is unpriced - and the cost of a second pass is the number that makes the case to whoever signs the invoice. If the dial you are reaching for is compression, trimming the token bill collects the techniques that do not change the architecture at all, and managing the context window covers how far a budget can be pushed before answers degrade.

How to tell which decision you got wrong

Symptoms are more reliable than opinions, and each of the common ones points at one decision.

Symptom The decision to revisit
Answers are thin, and the right document exists in the corpus Depth, and the index's chunk boundaries
Answers are confident and wrong, quoting nothing No score threshold, and no reranker between retrieval and prompt
The same question answers differently in two surfaces Two read paths with their own defaults
Cost grows while traffic is flat Compression off, or a loop with no iteration cap
One team sees another team's facts The filter was built in the caller instead of the edge of the stack
A document was updated and the answer did not move Index drift: nothing reconciles the index against the source

The last row is the one teams live with longest, because a stale answer is indistinguishable from a correct one until somebody checks the source. Reconciliation is cheap to add and easy to forget: one job that compares the listing against the store, one log line per dropped item, and the failure becomes visible the day it starts.

Most of those symptoms have a published counterpart: the surveys catalogued in the agentic rag survey describe the same failure modes under their own headings, which is a useful cross-check when a symptom has no obvious owner.

How SmartGate compares

The decisions above are the same for every stack; the difference is how many of them you get to make, and how many arrive as defaults you inherit.

The decisions it makes for you The decisions it leaves with you
A framework how boxes connect and how retrieval is called budgets, filters, limits and the loop's caps
A managed vector service the store, its scale and its filters chunking, reranking, compression and tenancy
A model provider's retrieval one index inside one vendor's world everything outside that vendor
SmartGate compression, deduplication, a hard budget guard, per-key limits and an audit row per call your corpus, your chunk boundaries and any index you keep

Handing over the middle of the stack is worth it because the middle is where the dials live. Free tier covers 2 million tokens a month with all seven tools and no card; Pro starts at 18 dollars a month and Teams at 55, with the same per-key limits on each plan.

How to get started

  1. Sort the decisions by reversibility. Re-chunking a corpus is a project; swapping a reranker is an afternoon. Spend the argument on the first group.
  2. Write the retriever's contract down. Score scale, filter language, depth default and the empty-result shape - four lines that decide whether the next swap is a deployment or a migration.
  3. Put identity in the filter at the edge. A caller-supplied tenant id is not a filter, it is a suggestion.
  4. Set the dials before you tune them. A cap on candidates, a cap on iterations and a budget check before a second pass.
  5. Reconcile the listing against the store on a schedule. One job, one log line per dropped item, and stale answers stop being invisible.
  6. Then read the diagram page. the diagram of these boxes prices every arrow these decisions move, and what an agentic loop adds is the version where the loop owns more of them.

Start on the free tier - 2 million tokens a month and all seven tools - from start free; the per-plan limits are on the pricing page, the endpoint shape is in the product documentation, and contract traffic starts at the contact form.

Frequently Asked Questions

Limitations and what this does not do

  • Two of the eight planned sections carry no code. The security boundary and the framework sections matched no unique symbol in the codebase, so they are written from published sources with no quoted implementation. Where a section shows no fence, that is the reason and not an omission.
  • This page argues; it does not price. Every dial in the table is described in structural terms. The money behind those dials is on the page this one points at, which takes the same stack apart in cost.
  • The excerpts show one stack's defaults. Twenty memories, a depth of ten candidates and a comma-separated configuration list are this implementation's choices, quoted because they are readable. Yours will differ, and the decision each one encodes does not.
  • The security section is a boundary argument, not a threat model. It says where identity has to be enforced and why retrieved text is untrusted input; it does not enumerate attacks against a specific deployment.
  • Nothing here replaces measuring your own corpus. Recall, latency and cost per query are properties of your data at your scale. The article tells you which dial to move; only your own evaluation tells you how far.

Sources

Method note

The code in this article is not transcribed. Each block was cut directly out of the slice body returned by the SmartGate slice API, then re-asserted byte for byte as a substring of that body before publication; the first line inside every fence records the file and the exact source lines. Symbols were pinned with whole-name containment (rule A level 2) and confirmed by the service's slot-proof endpoint - 5 of 8 planned sections pinned, no abstentions, and the three sections that matched no unique symbol are written from published sources with no code, as the abstain rule requires. Where a block shows two line windows, the gap between them is the part of the function that does not illustrate the decision being discussed, so it is not quoted. The excerpts come from one implementation of the stack, chosen because the decisions above exist there as code rather than as commentary.

Slice provenance

# SERP keyword Symbol File Source lines How it was pinned sha256(12)
1 mcp resources validateIndexItems lib/forgenova-kv-index.server.ts 22–43 rule A L2 → slot-proof 2a4af81b6244
2 openai mcp AsyncMemory.get_all backend/smartgate/modules/memory/algorithm.py 2431–2445, 2476–2492 rule A L2 → slot-proof b3cbbfac83d3
3 openclaw config parseConfigList lib/sitemap-config.ts 3–9 rule A L2 → slot-proof 5d57083f7b4e
4 vscode mcp convert backend/smartgate/modules/fetch/html_converter.py 82–97 rule A L2 → slot-proof d858200a75c5
5 agent2agent search_vectors backend/smartgate/core/resources.py 141–149 rule A L2 → slot-proof 0e5cc4aeeb5b

Every fenced block above was cut from the slice body and re-asserted against it byte for byte before publication. 5 of 8 sections pinned, 0 abstentions, 3 misses.