SmartGateSmartGate

RAG vs Agentic RAG: Choosing per Workload, Not in General

RAG and agentic RAG are not successive generations of one idea; they are the same retrieval stages driven by different control structures. Classic RAG fixes the query before the model runs — one search, one prompt, one answer, bounded latency and bounded cost.

Short answer: RAG and agentic RAG are not successive generations of one idea; they are the same retrieval stages driven by different control structures. Classic RAG fixes the query before the model runs — one search, one prompt, one answer, bounded latency and bounded cost. Agentic RAG lets the model issue the next query after reading the first result, which buys accuracy on multi-document questions and pays for it in calls, latency and an unbounded bill unless something enforces a ceiling. The decision is per workload: the same system should use both, routed by question complexity.

Key takeaways

  • The difference is control, not technology. Both paths embed, search and generate. One decides once; the other decides after every retrieval.
  • Latency is sequential in both cases, but the loop's sequence is longer. A second retrieval cannot start until the first result has been read and judged.
  • Cost per answer replaces cost per call as the unit. A loop that answers correctly in three calls can be cheaper than a pipeline that needs three attempts at a better prompt.
  • Failure modes are different, which is why the choice is not a ranking. Classic RAG cannot recover from a passage outside the top-k; agentic RAG can, and adds a runaway risk that a pipeline does not have.
  • Route by complexity. Question complexity is measurable before generation, which is exactly what makes routing — rather than a global preference — the practical answer.
  • Do this next: take your last twenty production questions, label each as lookup or synthesis, and write down how many of them a single retrieval could have answered. That ratio decides the architecture.

The retrieve-then-generate protocol: one query, one prompt

The classic path is a fixed sequence with one decision in it. The query is embedded, the index returns the top-k passages, the prompt is assembled from those passages, the model generates, and the result is rendered. Every step before generation is deterministic — the same question returns the same passages — and the final transformation is a pure function of the model's output:

# lib/pseo/markdown.ts — source lines 29–36 (pipelineHtmlFragment)
async function pipelineHtmlFragment(source: string): Promise<string> {
  const file = await unified()
    .use(rehypeParse, { fragment: true })
    .use(rehypeSanitize, sanitizeSchema)
    .use(rehypeStringify)
    .process(source);
  return String(file);
}

That is a sanitising renderer: markup in, safe fragment out, no branch that depends on the content. The property is worth naming because it is what a classic pipeline buys you: a path whose work and duration are known before the request starts. The original retrieval-augmented generation paper describes the same shape — a retriever feeding a generator — and the reason it became the default is that it is easy to operate, easy to cache and easy to evaluate. Its single structural weakness follows from the same property: the retrieval decision was made before the model saw anything, so a passage that is not in the top-k cannot be recovered by any later step.

The loop's protocol: a call ceiling and a fail-open limit

Move the query decision inside the run and the number of calls per answer stops being a property of the pipeline and becomes a property of the traffic. That is the point at which a rate limiter stops being infrastructure and becomes part of the answer's design:

# backend/smartgate/core/rate_limiter.py — source lines 51–91 (check_rate_limit_mcp)
async def check_rate_limit_mcp(
    key_id: str,
    team_id: str,
    *,
    per_key_limit: int,
    team_ceiling: int,
    window_seconds: int = 60,
) -> dict:
    try:
        redis = await get_redis()
    except Exception as exc:
        logger.warning("mcp rate_limit fail-open: %s", exc)
        return {"allowed": True, "fail_open": True}

    try:
        bucket = int(time.time() / window_seconds)
        key_bucket = f"rate_limit_mcp:{key_id}:{bucket}"
        team_bucket = f"rate_limit_mcp_team:{team_id}:{bucket}"

        key_count = await _incr_bucket(redis, key_bucket, window_seconds)
        if key_count > per_key_limit:
            ttl = await redis.ttl(key_bucket)
            return {
                "allowed": False,
                "retry_after": max(1, ttl),
                "limit_scope": "mcp_key",
            }

        team_count = await _incr_bucket(redis, team_bucket, window_seconds)
        if team_count > team_ceiling:
            ttl = await redis.ttl(team_bucket)
            return {
                "allowed": False,
                "retry_after": max(1, ttl),
                "limit_scope": "mcp_team",
            }

        return {"allowed": True}
    except Exception as exc:
        logger.warning("mcp rate_limit fail-open: %s", exc)
        return {"allowed": True, "fail_open": True}

Three things in that function are the actual comparison. It enforces two limits — one per credential, one for the whole team — because a loop that retries will exhaust a per-key allowance while a team ceiling is what stops one workload from starving the others. It returns a structured refusal with a retry interval and a scope, so the caller can tell "you are over your key's budget" from "the team is out", which are different operational problems. And it fails open: when the datastore is unreachable, the call is allowed with a flag set, on the reasoning that refusing every request during a limiter outage is worse than briefly serving unmetered traffic.

That last choice is the one to argue about, and the argument changes with the architecture. A pipeline issues one or two provider calls per question, so a burst is roughly proportional to user traffic. A loop multiplies every request by its calls per answer, so a caching regression, a prompt change and a genuine traffic spike produce the same shape on a dashboard. The consequence is not that the limiter should be stricter; it is that per-call attribution is what makes the graph readable — which is the same reason agent execution cost is measured per call rather than per question.

The comparison protocol: rows from one catalog

A comparison is only as trustworthy as its table, and a hand-written table drifts from the thing it describes. The function below builds the rows from a single plan catalog and the rate-limit accessors, so the published table and the enforced limits cannot disagree:

# config/marketing-pricing.ts — source lines 98–142 (getMarketingCompareRows)
function getMarketingCompareRows(): MarketingCompareRow[] {
  return [
    {
      feature: "Monthly token cap (L1)",
      values: Object.fromEntries(
        MARKETING_PLAN_ORDER.map((plan) => [plan, formatTokenCap(plan)]),
      ) as Record<MarketingPlanId, CompareCell>,
    },
    {
      feature: "Full MCP tools",
      values: { FREE: true, PRO: true, TEAMS: true, ENTERPRISE: true },
    },
    {
      feature: "Activity log retention",
      values: Object.fromEntries(
        MARKETING_PLAN_ORDER.map((plan) => [
          plan,
          `${PLAN_CATALOG[plan].features.activity_log_retention_days}d`,
        ]),
      ) as Record<MarketingPlanId, CompareCell>,
    },
    capabilityRow("L2 hard cap (β)", "l2BetaAllowed"),
    capabilityRow("Member tool overrides", "memberToolOverridesAllowed"),
    capabilityRow("Team gateway policy", "teamGatewayPolicyEditable"),
    capabilityRow("Integrator HMAC budget", "l2HmacBudgetAllowed"),
    {
      feature: "Max API keys",
      values: Object.fromEntries(
        MARKETING_PLAN_ORDER.map((plan) => [
          plan,
          String(PLAN_CATALOG[plan].features.max_api_keys),
        ]),
      ) as Record<MarketingPlanId, CompareCell>,
    },
    rpmRow("MCP req/min (per API key)", (p) =>
      String(getPlanRateLimits(p).mcpRpmPerKey),
    ),
    rpmRow("REST write req/min (team)", (p) =>
      String(getPlanRateLimits(p).restWriteRpm),
    ),
    rpmRow("MCP team ceiling (req/min)", (p) =>
      String(getPlanRateLimits(p).mcpRpmTeamCeiling),
    ),
  ];
}

Read the row names as the checklist, then substitute the objects. A cap appears as a token allowance per month; here it would be the context and call budget per question. A capability list appears as which tools a plan includes; here it is which tools each path may call — a pipeline gets one search, a loop gets a search it can repeat. A retention figure appears as log days; here it is how long the loop's state survives and whether the next question can see it. Per-plan overrides and per-minute limits have direct analogues in ceilings, retries and provider throttling. Building both tables from one catalog is also the cheapest way to keep this page honest: change the ceiling and the row changes with it.

Tools: what each path is allowed to call

The sharpest structural difference is the tool surface. A classic pipeline needs two operations — retrieve and generate — and often only one, since the retriever can be a library call. An agentic loop is defined by a surface it can choose from, and in a hosted implementation each tool is declared rather than hard-coded:

# backend/smartgate/api/mcp.py — source lines 106–148 (register_mcp_tools)
def register_mcp_tools(server: FastMCP) -> None:
    """Register all 7 smart_* tools on a FastMCP instance."""

    @server.tool(
        name="smart_fetch",
        description=TOOL_DESCRIPTIONS["smart_fetch"],
        annotations=tool_annotations("smart_fetch"),
    )
    async def smart_fetch(
        url: str = Field(description="Full HTTP or HTTPS URL to fetch."),
        timeout: int = Field(default=30, description="HTTP timeout in seconds."),
    ) -> str:
        _, registry = _app_state()
        module = registry.get("fetch")
        ctx = _tool_ctx()
        return await _run_with_audit(
            "fetch",
            ctx,
            module.process(ctx, url=url, timeout=timeout),
            {"url": url},
        )

    @server.tool(
        name="smart_search",
        description=TOOL_DESCRIPTIONS["smart_search"],
        annotations=tool_annotations("smart_search"),
    )
    async def smart_search(
        query: str = Field(description="Search query string."),
        max_results: int = Field(
            default=10,
            description="Maximum number of results to return (1–50).",
        ),
    ) -> str:
        _, registry = _app_state()
        module = registry.get("search")
        ctx = _tool_ctx()
        return await _run_with_audit(
            "search",
            ctx,
            module.process(ctx, query=query, max_results=max_results),
            {"query": query},
        )

# backend/smartgate/api/mcp.py — source lines 150–174 (register_mcp_tools)
    @server.tool(
        name="smart_context_gate",
        description=TOOL_DESCRIPTIONS["smart_context_gate"],
        annotations=tool_annotations("smart_context_gate"),
    )
    async def smart_context_gate(
        text: str = Field(description="Long text to compress before the host LLM call."),
        ratio: float = Field(
            default=0.5,
            description="Target compression ratio (e.g. 0.3–0.7).",
        ),
        purpose: str | None = Field(
            default=None,
            description="Optional goal to pre-filter paragraphs (step intent, user query).",
        ),
    ) -> str:
        _, registry = _app_state()
        module = registry.get("context_gate")
        ctx = _tool_ctx()
        return await _run_with_audit(
            "compress",
            ctx,
            module.process(ctx, text=text, ratio=ratio, purpose=purpose),
            {"ratio": ratio, "purpose": purpose},
        )

Every registration is a degree of freedom with a price. The smart_fetch tool takes a URL and a timeout; smart_search takes a query and a result count; the compression tool takes long text before the host model call, which is the one that changes the arithmetic of a loop, because it reduces the per-call context that every subsequent hop re-sends. The annotations deserve the same attention as the signatures: a read-only hint is what lets a host decide which calls may run without a human in the path, and a tool without one is a tool a careful runtime will not let an unattended agent invoke. The audit wrapper at the end of each registration is the part that makes the comparison measurable — every call records its tool name and parameters, so "how many retrievals did this answer take" is a query rather than an estimate.

The practical rule: a workload that needs one tool should not be given five. The surface is where an agentic system's cost variance comes from. Token optimization for apps covers the compression side, and the shared gateway is where the surface gets its limits.

Latency: what the second call costs

Latency is where the loop's advantage is thinnest. A classic request is two sequential stages — retrieve, then generate — and the retrieval stage is normally the cheap one, which is why streaming makes a pipeline feel fast: the first tokens arrive while the rest of the answer is still being produced. An agentic loop inserts a judgement between stages that cannot be streamed away: the model has to read the retrieved passages and decide before the next query is issued, and that decision is a generation in its own right.

Path Stages per answer What dominates p50 What dominates p95
Retrieve-then-generate 1 retrieval + 1 generation Generation length Provider queueing on the generation call
Loop, two hops 2 retrievals + 2 judgements + 1 generation The judgements The slowest of the retrievals
Loop, five hops 5 retrievals + 5 judgements + 1 generation The judgements Whatever call the provider throttles last

Two mitigations are honest and one is not. Honest: cache the first retrieval, because the first query of a question type repeats across users; and cap the hops, because latency grows with the count and the later hops have the lowest marginal accuracy. Not honest: promising the pipeline's p95 while running a loop behind the same endpoint. Where the question complexity is knowable in advance, routing at that point is the technique that makes both numbers defensible (complexity routing).

Cost: from tokens per call to calls per answer

Cost comparisons go wrong when the unit is wrong. A pipeline and a loop do not differ much in the price of one call; they differ in how many calls the same question needs, and in how much context each call carries. Three quantities decide the outcome, and all three are measurable before you choose: the number of retrieval calls per accepted answer, the mean context per call, and the retry rate. A loop with three calls and 2,000 tokens of context each is a different product from a loop with two calls and 30,000 tokens each — the second is where compression and deduplication earn their place, and where the context budget becomes the binding constraint rather than the retrieval count.

The counter-effect to keep in view is that each hop re-sends the accumulated state. A loop that writes its scratchpad into every prompt pays for the same passages once per hop, so an implementation that retrieves conservatively and compresses aggressively often beats one that retrieves widely and relies on the model to ignore what it does not need. Measure cost per accepted answer, not per call: a pipeline that needs three attempts at prompt tuning to answer a synthesis question can cost more than a loop that answers it in two retrievals and stops.

Where each fails

Failure modes are the reason this is a routing decision rather than a ranking. Each path has one class of question it cannot answer and one way it degrades badly.

Failure Retrieve-then-generate Agentic RAG
The needed passage is not in the top-k Unrecoverable — no later step can fetch it Recoverable — the next query is written with knowledge of the miss
Multi-document synthesis (compare, total, reconcile) Usually wrong, confidently Handled, at the cost of several retrievals
Unanswerable question Answers anyway, because the prompt contains something Can abstain if a judge is asked to decide
Latency target under an SLO Predictable, easy to cache Grows with hops; the cap is the design
Unattended traffic Bounded by request count Unbounded unless the ceiling and the budget are enforced
Evaluation and regression testing Reproducible — same query, same passages Non-deterministic — the query is chosen at run time

The last row is underrated. A loop is harder to test than a pipeline, not because the components are worse but because the path through them is a model output. Teams that adopt a loop without recording its calls per answer lose the ability to tell a retrieval regression from a reasoning one, which is why the audit row is a prerequisite rather than an observability nicety.

Choosing per workload

The decision is a small set of rules, and it is worth writing them down rather than arguing about preferences per feature request.

  • One lookup, one passage, cached answers — classic. A comparison, a total or a policy lookup does not get better with a second query, and the pipeline's determinism is worth more than the accuracy a loop might add.
  • Synthesis across documents — agentic, with a hop ceiling of two or three. This is the workload the loop exists for, and the marginal value of the second and third query is highest here.
  • Strict latency target with no budget — classic, or a loop capped at two hops. Anything longer than that is a latency promise you will break under load.
  • Mixed traffic — route by complexity, not by preference: classify the question cheaply, send lookups down the pipeline and synthesis questions into the loop.
  • Unattended and high-volume — agentic only with a call ceiling, a spend limit and per-call records in place. Without all three, a routing bug becomes an invoice.
  • Anything you cannot evaluate — classic. The deterministic path is the baseline that lets you prove the loop is an improvement rather than an assumption.

The stages both paths share are documented on the architecture both routes share, and what the loop is works through the mechanism behind the second column. The two siblings that frame this comparison are the shared layer diagram and the research landscape.

Two more pages sit either side of that frame: rag vs agentic ai separates the retrieval pattern from the agentic runtime, and agentic search vs RAG works out the version where the retrieval path is somebody else's search engine.

How SmartGate compares

The choice is not between two algorithms; it is where the ceiling, the credentials and the record of each call live.

What the retrieval decision is What it costs Where the controls live
A library pipeline (retriever plus generator) Fixed before the model runs One retrieval and one generation per answer In your code; nothing enforces a ceiling
A framework graph (LangChain, LangGraph) A node's output, routed by a conditional edge Your infrastructure plus what you meter In the graph's configuration, in process
A bespoke loop with a proxy The model's next action Whatever the proxy fails to count Wherever the team put them, usually two places
SmartGate The model chooses from seven declared tools Per-call, with per-key rate limits and a hard budget guard One endpoint: limits, budget and an audit row per call

The row that decides most comparisons is the last column. A ceiling that lives somewhere other than the call path is a ceiling that disagrees with the meter, and the disagreement shows up as a budget overrun rather than an error. Free tier: 2M tokens a month, all seven tools. Pro from $18 a month, Teams from $55 a month. The memory design covers what a loop keeps between hops; this page's point is only that the controls belong on the same path as the calls they govern.

How to get started

  1. Label twenty production questions. Lookup or synthesis. Do not skip this step — the ratio is the only input the decision needs, and most teams find the lookup share is larger than they assumed.
  2. Measure the pipeline's baseline first. Retrieval calls, context per call, and accepted answers on those twenty questions. A loop compared against an intuition is a loop compared against nothing.
  3. Run the synthesis questions as a loop by hand. Two hops, written queries, and note which second query changed the answer. That is your per-hop marginal accuracy.
  4. Set the two ceilings before the first automated run. Maximum hops per question and a spend limit that refuses rather than warns. Both are configuration; neither can be added after an incident.
  5. Route, then measure both sides separately. Keep one metric per path, because a blended average hides exactly the regression you are watching for.
  6. Re-decide when the workload changes. A workload that was all synthesis last quarter is often all lookups once the cached answers exist, and the ratio is what should move the routing.

Start on the free tier — 2 million tokens a month and all seven tools — with start free; the per-plan limits are on the pricing page, the tool parameters are in the product docs, and contract traffic starts at the contact form.

Frequently Asked Questions

Limitations and what this does not do

  • Four of the eight planned sections carry code. The latency, cost, failure-mode and workload-routing sections matched no unique symbol in this codebase and are written from the published research with no quoted implementation.
  • The tool surface is quoted through windows. Two of the seven registrations are shown; the rest repeat the same shape and add nothing to the argument about the surface's size.
  • The latency table is a shape, not a measurement. It names what dominates each percentile so you can instrument your own path. It reports no timings, because the timings that matter are the ones on your deployment.
  • No single rule replaces a test. The routing rules are the decision's skeleton; the ratio of lookup to synthesis questions in your traffic is what fills it in, and it changes as your cached answers accumulate.
  • This page is a comparison, not a tutorial. It does not show how to build either path — the loop's mechanism and the retrieval stages underneath each have their own page in the cluster.

Sources

Method note

The code in this article is not transcribed. Each block was cut directly out of the slice body returned by the SmartGate slice API and re-asserted byte-for-byte as a substring of that body before publication; the first line inside every fence records the file and the exact source lines. Symbols were pinned with whole-name containment (rule A level 2) and confirmed by the service's slot-proof endpoint — 4 of 8 planned sections pinned, no abstentions. Four sections (latency, cost, failure modes and workload routing) matched no unique symbol and are written from the published research with no code, as the abstain rule requires. The quoted implementation is the gateway's own rendering, rate-limiting, comparison-row and tool-registration code, used as the concrete form of the controls a per-workload decision has to place somewhere.

Slice provenance

# SERP keyword Symbol File Source lines How it was pinned sha256(12)
1 model context protocol pipelineHtmlFragment lib/pseo/markdown.ts 29–36 rule A L2 → slot-proof d00855a0e918
2 mcp protocol check_rate_limit_mcp backend/smartgate/core/rate_limiter.py 51–91 rule A L2 → slot-proof b540f8154beb
3 a2a protocol getMarketingCompareRows config/marketing-pricing.ts 98–142 rule A L2 → slot-proof 7c64410591e2
4 mcp tools register_mcp_tools backend/smartgate/api/mcp.py 106–148, 150–174 rule A L2 → slot-proof 9d4a1623b28c

Every fenced block above was cut from the slice body and re-asserted against it byte-for-byte before publication. 4 of 8 sections pinned, 0 abstentions, and 4 section(s) written from external sources because no unique symbol matched.