What Is Agentic RAG? Retrieval as a Loop, Not a Pipeline

Short answer: Agentic RAG is retrieval an agent drives instead of a pipeline that runs once: the model sends a search, reads what came back, judges whether that is enough, and either answers or searches again with a better query. Nothing about the change requires a larger model or a larger index — the number of retrieval calls and the shape of each query become decisions made at run time, and a loop that decides at run time needs somewhere to keep what it has learned, a condition that stops it, and a bill you can attribute to a tool.

Key takeaways

• The pipeline became a control problem. Retrieve-then-generate fixes the query before the model sees anything. The loop makes the query an output, so the same question can cost one retrieval call or twelve.
• State is the part teams skip. A loop that cannot write down what it found will re-issue the query it just ran; a loop that cannot prune will carry every dead end into the final prompt.
• Stopping is a feature. The budget, the call ceiling and the confidence threshold are what turn a loop into a product instead of an invoice.
• Cost moves from tokens per call to calls per answer. Which tool spent the tokens is a measurable, rankable fact rather than an estimate, if the loop records it.
• Frameworks express this as a graph plus configuration. LangChain, LangGraph and the rest ask you to declare nodes, edges and a state schema — the loop's parameters live in that declaration.
• Do this next: take one question your system answers badly, run it by hand as retrieve → judge → retrieve, and write down how many calls and how many distinct queries it actually took. That number is the budget your implementation has to beat.

---

What changes when retrieval becomes a loop

A retrieve-then-generate pipeline decides everything about retrieval before the model runs. One query string goes to the index, the top-k passages come back, and the prompt is assembled from whatever arrived — the shape Lewis and colleagues described in the original RAG paper. The loop moves the same decisions inside the run. Whether to retrieve at all, what to search for next, and whether the passages that came back are sufficient are now outputs of the model rather than inputs to it. That is the whole idea, and it is older than the phrase: ReAct interleaved reasoning traces with actions (reasoning and acting), and IRCoT interleaved retrieval with chain-of-thought steps (interleaved retrieval).

Three things become runtime variables, and each one has a cost attached:

• Whether to retrieve. A greeting, a summarisation of text already in the prompt and a question about a private document do not deserve the same treatment.
• What to query. The first query is a guess; the second query is written with knowledge of what the first one returned, which is where the accuracy gain comes from.
• Whether to accept. A loop that cannot decide it is finished does not stop, and an unattended loop does not warn before it bills.

The rest of this page works through what those three variables demand from an implementation. If you want the field-level view of the same shift first, the agentic RAG research survey groups the published work by which of the three variables each method attacks, and the layer diagram shows where each stage sits in a deployed stack. The question this page answers is narrower: what does it actually take to run retrieval as a loop, and what does the loop lean on that a pipeline never needed? The stages underneath — indexing, deduplication, compression, retrieval — are on the retrieval architecture in full, which is the page to read when the loop is working and the passages it fetches are wrong.

The memory architecture a loop needs

A single-pass pipeline is stateless between requests, which is why it is easy to operate. A loop is stateful for the duration of a question, and sometimes across questions. Three kinds of state show up in every working implementation. First, a scratchpad: the claims the agent has established so far, each one tied to the passage and the query that produced it, and it has to be small enough to fit in the prompt next to the retrieved passages. Second, a record of attempts: the queries already issued and what they returned, so the agent does not spend its second call repeating its first. Third, the identity the question is about — a user, a document, a ticket — because retrieving the correct passage for the wrong entity is the most common way a loop produces a confident wrong answer.

Getting that state into the prompt is its own problem. Passage position matters more than most teams expect (long-context behaviour), and a scratchpad that grows with every hop competes with the passages for the same window — context window management covers the accounting. The practical consequence is that the state store and the retrieval index are different systems with different jobs, and an architecture that treats the scratchpad as "just another vector collection" usually discovers the difference when the loop reads a stale note.

Memory management: the write path after every hop

The habit that separates a loop that converges from one that spirals is writing the result of each hop back into the state store. In the implementation quoted below, that write is an update by identifier rather than an append:

# backend/smartgate/modules/memory/algorithm.py — source lines 2909–2931 (AsyncMemory.update)
async def update(self, memory_id, data, metadata: Optional[Dict[str, Any]] = None):
        """
        Update a memory by ID asynchronously.

        Args:
            memory_id (str): ID of the memory to update.
            data (str): New content to update the memory with.
            metadata (dict, optional): Metadata to update with the memory. Defaults to None.

        Returns:
            dict: Success message indicating the memory was updated.

        Example:
            >>> await m.update(memory_id="mem_123", data="Likes to play tennis on weekends")
            {'message': 'Memory updated successfully!'}
        """
        capture_event("mem0.update", self, {"memory_id": memory_id, "sync_type": "async"})

        embeddings = await asyncio.to_thread(self.embedding_model.embed, data, "update")
        existing_embeddings = {data: embeddings}

        await self._update_memory(memory_id, data, existing_embeddings, metadata)
        return {"message": "Memory updated successfully!"}

Three details carry real weight. The update takes an identifier, so the agent is revising a note it already chose rather than creating a second one; a store that only appends forces the loop to deduplicate later, at query time, where it is more expensive and harder to debug. The new content is re-embedded before the write — the note's text changed, so the vector has to change with it, and an implementation that skips that step leaves a memory whose similarity score describes something the note no longer says. And the write is asynchronous, which matters because the loop is already waiting: a synchronous round trip to an embedding service inside a hop is latency the answer did not need.

The failure mode to design against is the update that succeeds twice. Nothing in that signature makes it idempotent, so a retry after a timeout writes the same note again with a fresh timestamp and a fresh embedding. If your loop retries, carry a request identifier into the metadata and treat the pair as the key. Teams that get this right early spend their debugging time on retrieval quality instead of on duplicate rows; teams that do not, discover it when a stakeholder asks why the same passage appears four times in one answer.

Tools and the token bill of a loop

A loop calls more than a retriever. Fetch, search, compress, deduplicate and memory all consume tokens, and the interesting question after a week in production is not the total but the split. The function below ranks the tools a real dashboard shows by what they spent, with a fallback:

# lib/analytics/top-tools.ts — source lines 14–47 (computeTopTools)
function computeTopTools(
  byTool: { tool: string; tokens: number; calls: number }[],
  limit = 3,
): TopToolRow[] {
  const productRows = filterProductAuditEntries(byTool);
  const rows = productRows.length > 0 ? productRows : byTool;
  if (rows.length === 0) return [];

  const totalTokens = rows.reduce((sum, row) => sum + row.tokens, 0);
  if (totalTokens > 0) {
    return [...rows]
      .sort((a, b) => b.tokens - a.tokens)
      .slice(0, limit)
      .map((row) => ({
        tool: row.tool,
        tokens: row.tokens,
        sharePct: Math.round((row.tokens / totalTokens) * 100),
        byCalls: false,
      }));
  }

  const totalCalls = rows.reduce((sum, row) => sum + row.calls, 0);
  if (totalCalls <= 0) return [];

  return [...rows]
    .sort((a, b) => b.calls - a.calls)
    .slice(0, limit)
    .map((row) => ({
      tool: row.tool,
      tokens: row.tokens,
      sharePct: Math.round((row.calls / totalCalls) * 100),
      byCalls: true,
    }));
}

The structure of that function is the argument. It totals tokens per tool and returns a ranked share, so the answer to "why was yesterday expensive" is a list of three names rather than a number. When token accounting is missing — an audit row written before metering existed, a tool that reports calls but not usage — it falls back to ranking by call count and marks the row, which means the panel degrades to a weaker but honest measure instead of showing nothing. That is a small decision with a large operational payoff: a ranking that quietly disappears when data is missing teaches operators to distrust the panel.

For an agentic loop the same ranking answers a different question. A retrieval-heavy question and a compression-heavy question look identical in a monthly total and completely different in that table, and only one of them is fixed by caching. If the loop calls its tools through a shared endpoint, per-key limits and an audit row make the split a query rather than a reconstruction — what a gateway adds is the longer version of that argument, and the cost of an agent loop works through the arithmetic of calls per answer.

One variant of that bill is worth naming separately: when the retrieval path is a search tool rather than an index you operate, the per-call cost is the whole story and the ranking is somebody else's — agentic search vs RAG counts that version.

Memory pruning: deleting the passage that did not help

Retrieval adds; answers require subtraction. A passage that looked relevant at hop one and turned out to be a dead end is worse than a passage never fetched, because the loop paid to embed it, paid to carry it in the prompt, and will pay again on the next hop if nothing removes it. The delete path below is what that subtraction looks like when the store is a vector collection with identifier semantics:

# backend/smartgate/modules/memory/algorithm.py — source lines 2933–2947 (AsyncMemory.delete)
async def delete(self, memory_id):
        """
        Delete a memory by ID asynchronously.

        Args:
            memory_id (str): ID of the memory to delete.
        """
        capture_event("mem0.delete", self, {"memory_id": memory_id, "sync_type": "async"})

        existing_memory = await asyncio.to_thread(self.vector_store.get, vector_id=memory_id)
        if existing_memory is None:
            raise ValueError(f"Memory with id {memory_id} not found")

        await self._delete_memory(memory_id, existing_memory)
        return {"message": "Memory deleted successfully!"}

Read it for the read-modify-write. The store is asked for the record before anything is deleted, because the deletion expands to every index that holds a copy — vector data, payload, history — and that expansion needs the record. When the identifier does not exist, the function raises rather than returning quietly. Inside a loop that is a decision you have to make deliberately: a missing identifier during a cleanup step usually means the state was already pruned, and an agent that propagates the exception dies in the middle of a question. Catching it and continuing is often right, but only if the log records the miss, because the silent version of this failure is a scratchpad that never shrinks.

Pruning also has a security dimension that memory-oriented frameworks tend to leave implicit. Data that should have expired may still be sitting in a collection that a later question will search, and the delete path is the only mechanism that removes it. Deciding how long a note lives is a product decision; enforcing it is this function.

Memory reset: one question, one workspace

Between questions, the loop's state is not an asset. Yesterday's notes about a different customer are actively harmful in today's prompt, and the cheapest way to guarantee they cannot appear is to destroy the workspace rather than to filter it. A full reset touches more than the vector index:

# backend/smartgate/modules/memory/algorithm.py — source lines 3188–3213 (AsyncMemory.reset)
async def reset(self):
        """
        Reset the memory store asynchronously by:
            Deletes the vector store collection
            Resets the database
            Recreates the vector store with a new client
        """
        logger.warning("Resetting all memories")
        await asyncio.to_thread(self.vector_store.delete_col)

        gc.collect()

        if hasattr(self.vector_store, "client") and hasattr(self.vector_store.client, "close"):
            await asyncio.to_thread(self.vector_store.client.close)

        if hasattr(self.db, "connection") and self.db.connection:
            await asyncio.to_thread(lambda: self.db.connection.execute("DROP TABLE IF EXISTS history"))
            await asyncio.to_thread(self.db.connection.close)

        self.db = SQLiteManager(self.config.history_db_path)

        self.vector_store = VectorStoreFactory.create(
            self.config.vector_store.provider, self.config.vector_store.config
        )

        capture_event("mem0.reset", self, {"sync_type": "async"})

The order matters, and it is the reason this is a function rather than a call to the store's own clear method. The collection is dropped first; the garbage collector is invoked because the client that held it is about to be replaced; the client is closed; the history table is dropped; and the store is recreated through the same factory that built it, so a fresh question gets a store with the same configuration as the first one. Anything the reset forgets becomes a leak that shows up as cross-contamination weeks later, when nobody remembers what changed.

Two costs are worth naming honestly. A hard reset throws away warm connections and any cached embeddings, so on a question-by-question workload it is not free; the alternative — logical isolation by user or thread key — keeps the process warm at the price of trusting that every query filters correctly. And a reset is a destructive operation with no undo, so it belongs behind an explicit action rather than a fallback path.

LangChain, LangGraph and the loop as configuration

Frameworks did not invent the loop; they made it declarable. In LangChain and LangGraph the agentic part is a graph — nodes for the model, the tools and the conditional edge that sends the run back around — and the state is a schema the graph carries between nodes. What is left at the bottom of that stack is construction from a dictionary, which is where most integration bugs actually live:

# backend/smartgate/modules/memory/algorithm.py — source lines 534–542 (Memory.from_config)
@classmethod
    def from_config(cls, config_dict: Dict[str, Any]):
        try:
            config = cls._process_config(config_dict)
            config = MemoryConfig(**config_dict)
        except ValidationError as e:
            logger.error(f"Configuration validation error: {e}")
            raise
        return cls(config)

The interesting line is the validation. The configuration is handed to a typed model and a failure raises immediately, at construction, rather than surfacing three hops later as a store that quietly fell back to an in-memory default. When you wire a loop in a framework, the same rule applies to your own code: validate the state schema and the tool configuration before the first question, not during it. A loop that cannot be built correctly should not start.

Two practical notes from the same fragment. The class method returns a configured instance rather than mutating a global, so two workloads can hold two stores with different settings — which is how you isolate a test question from a production one. And the configuration is data: it belongs in the same review process as the rest of your deployment inputs. If your loop runs on token optimization budgets, the retrieval parameters and the compression ratio are inputs to the same decision, so keep them in one file.

The vocabulary is worth settling before the next decision, because the sentence "this loop is RAG" has to be checkable: rag vs agentic ai separates the retrieval pattern from the agentic runtime, and the two are bounded differently.

How to benchmark a loop

The benchmark question for agentic RAG is not "is the answer better" but "better by how much, for how many extra calls". Three published instruments are useful as templates rather than as scoreboards. GAIA (assistant benchmark) asks multi-step questions whose answers require tool use, which makes it a test of the loop rather than of the retriever. RAGAS (reference-free evaluation) measures the retrieval and generation halves separately, which is what tells you whether a weak answer is a retrieval problem or a reasoning problem. Search-R1 (learned search use) trains a model to interleave searches and reasoning, and its evaluation records the number of searches per question as a first-class result rather than an implementation detail.

Record four numbers per question, not one: the number of retrieval calls, the number of distinct queries, the total tokens, and whether the answer was accepted. The pair that matters is calls and acceptance — a loop that gains five accuracy points for one extra call is a different product from one that gains six for eleven extra calls. A benchmark table that reports only the accuracy column hides the trade the reader has to make. Retrofitting the call count is not hard if the audit rows already exist; it is impossible if the loop only logs its final answer. The trade cuts both ways, though, and the classic-versus-agentic comparison works the same calls-versus-accuracy arithmetic through latency, cost and answer shape for the workloads where a single retrieval is still the cheaper answer.

How SmartGate compares

The choice is not between one loop and another; it is about where the identity, the limit and the record of each call live.

	What it decides	Where the state lives	What it costs
A retrieve-then-generate pipeline	The query, before the model runs	Nowhere between requests	One embedding and one generation per question
A framework graph (LangChain, LangGraph)	The route through the graph, declared in code	In the graph's state schema, in process	Your infrastructure, plus what you build to meter it
A hand-rolled loop	Everything, in code you maintain	In whatever store you wired first	Maintenance, and the failure modes nobody priced
SmartGate	The tool surface the loop calls, with per-key limits, a hard budget guard and an audit row per call	In the gateway's memory tool, scoped to your key	Free tier: 2M tokens a month and all seven tools. Pro from $18 a month, Teams from $55 a month

The difference that shows up first in production is the last column's granularity. A loop calls tools a dozen times per question, and a system that records one row per question cannot answer which call was expensive. agent memory architecture covers what the store keeps; this page's point is narrower — the loop needs a record it can query, and a record that only exists in an application log is a record you will reconstruct by hand.

How to get started

1. Answer one hard question by hand. Pick something that needs two sources — a comparison, a total, a decision. Run retrieve, then judge, then retrieve again and write down each query. Most teams find they wrote three queries and their system writes one.
2. Count the calls before changing anything. Instrument the existing path so the number of retrievals per question is recorded. Without a baseline, every later claim about the loop is an impression.
3. Give the loop a scratchpad with identifiers. Malformed values are the usual failure here: an update path that keys on content creates duplicates the moment the note is rephrased.
4. Set the two ceilings you cannot bootstrap later. A maximum number of retrieval calls per question and a spend limit that stops the run. Both are configuration, and both are cheaper to add before the first unattended run than after it.
5. Prune on a schedule, and reset between questions. Deletion is the only mechanism that removes data from a searchable collection, and isolation is the only mechanism that stops one question's notes from answering the next one.
6. Then measure the loop against the baseline. Calls, tokens, accepted answers. If the loop does not beat the pipeline on the question you picked, it will not beat it on your workload.

Start on the free tier — 2 million tokens a month and all seven tools — with start free; the per-plan limits are on the pricing page, the tool parameters are in the product docs, and contract traffic starts at the contact form.

Limitations and what this does not do

• Five of the eight planned sections carry code. The section on what the loop changes, the section on the state it needs and the benchmarking section matched no unique symbol in this codebase, so they are written from the published literature with no quoted implementation.
• The code is one memory implementation, not a reference loop. The slices show the write, delete, reset and construction paths of a store with identifier semantics. They do not show the agent, the graph or the stopping logic — a loop is more than its storage.
• The benchmark instruments are templates, not a scoreboard. GAIA, RAGAS and Search-R1 measure different things on different question sets. Use them to choose which number to record, not to claim a win.
• A budget guard is a ceiling, not a plan. It stops the run that would have cost you an afternoon of explaining. It does not make a badly designed loop cheap, and a loop that hits the ceiling every week is a loop whose retrieval needs fixing.
• This page is not a substitute for the architecture page. It covers what changes when retrieval becomes a loop; the index, dedup and compression stages of a retrieval stack are documented on the cluster hub.

Sources

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
• Yao et al. — ReAct: Synergizing Reasoning and Acting in Language Models: https://arxiv.org/abs/2210.03629
• Trivedi et al. — Interleaving Retrieval with Chain-of-Thought Reasoning (IRCoT): https://arxiv.org/abs/2212.10509
• Liu et al. — Lost in the Middle: How Language Models Use Long Contexts: https://arxiv.org/abs/2307.03172
• Mialon et al. — GAIA: a benchmark for General AI Assistants: https://arxiv.org/abs/2311.12983
• Es et al. — RAGAS: Automated Evaluation of Retrieval Augmented Generation: https://arxiv.org/abs/2309.15217
• Jin et al. — Search-R1: Training LLMs to Reason and Leverage Search Engines: https://arxiv.org/abs/2503.09516
• SmartGate — documentation, pricing and sales contact: https://smartgate.network/docs · https://smartgate.network/pricing · https://smartgate.network/contact