Agent Memory Architecture: What to Store and Retrieve

Short answer: An agent memory architecture is three decisions rather than one library: what is written as a vector plus a payload, when a retrieval call is allowed to run, and what each of those two paths costs. SmartGate exposes the pattern as a single MCP tool, so the host owns the decision and the gateway owns the storage, the scoping and the accounting.

Key takeaways

• Memory is two paths, not one. A write pays for an embedding and a vector upsert; a read pays for a similarity search. They fail differently and they are optimized differently.
• Scoping is the safety model. Retrieval only searches inside the scope you pass — user, agent or run — and a call that names no scope is rejected before it reaches the store.
• Retrieval is a threshold, not a top-N list. A score floor plus a result cap is what stops a plausible-but-wrong memory from entering the context window.
• Cost is a background-worker question. Whatever keeps memory alive between calls is a process; a worker that is not stopped keeps paying for nothing.
• Exposure is one line in tools/list and one file per host. The interesting engineering is behind the endpoint, not in the config block.
• Measure the hit rate before you buy storage. A store that answers 95% of retrievals needs better search, not more vectors.

---

Semantic retrieval: the read path over agent memory

Retrieval is the half of memory that a context window actually feels. An agent calls search with a query, a scope and a score floor, and gets back the memories that cleared the floor. The entry point is one method, and most of its arguments are decisions about scope:

# backend/smartgate/modules/memory/algorithm.py — source lines 1126–1237 (Memory.search)
def search(
        self,
        query: str,
        *,
        top_k: int = 20,
        filters: Optional[Dict[str, Any]] = None,
        threshold: float = 0.1,
        rerank: bool = False,
        **kwargs,
    ):
        """
        Searches for memories based on a query.

        Args:
            query (str): Query to search for.
            top_k (int, optional): Maximum number of results to return. Defaults to 20.
            filters (dict): Filter dict containing entity IDs and optional metadata filters.
                Must contain at least one of: user_id, agent_id, run_id.
                Example: filters={"user_id": "u1", "agent_id": "a1"}

                Enhanced metadata filtering with operators:
                - {"key": "value"} - exact match
                - {"key": {"eq": "value"}} - equals
                - {"key": {"ne": "value"}} - not equals
                - {"key": {"in": ["val1", "val2"]}} - in list
                - {"key": {"nin": ["val1", "val2"]}} - not in list
                - {"key": {"gt": 10}} - greater than
                - {"key": {"gte": 10}} - greater than or equal
                - {"key": {"lt": 10}} - less than
                - {"key": {"lte": 10}} - less than or equal
                - {"key": {"contains": "text"}} - contains text
                - {"key": {"icontains": "text"}} - case-insensitive contains
                - {"key": "*"} - wildcard match (any value)
                - {"AND": [filter1, filter2]} - logical AND
                - {"OR": [filter1, filter2]} - logical OR
                - {"NOT": [filter1]} - logical NOT
            threshold (float, optional): Minimum score for a memory to be included. Defaults to 0.1.
            rerank (bool, optional): Whether to rerank results. Defaults to False.

        Returns:
            dict: A dictionary containing the search results under a "results" key.
                  Example for v1.1+: `{"results": [{"id": "...", "memory": "...", "score": 0.8, ...}]}`

        Raises:
            ValueError: If filters doesn't contain at least one of user_id, agent_id, run_id,
                or if threshold/top_k values are invalid.
        """
        # Reject top-level entity params - must use filters instead
        _reject_top_level_entity_params(kwargs, "search")

        # Validate search parameters (before applying defaults)
        _validate_search_params(threshold=threshold, top_k=top_k)

        # Validate and trim entity IDs in filters
        effective_filters = filters.copy() if filters else {}
        if "user_id" in effective_filters:
            effective_filters["user_id"] = _validate_and_trim_entity_id(
                effective_filters["user_id"], "user_id"
            )
        if "agent_id" in effective_filters:
            effective_filters["agent_id"] = _validate_and_trim_entity_id(
                effective_filters["agent_id"], "agent_id"
            )
        if "run_id" in effective_filters:
            effective_filters["run_id"] = _validate_and_trim_entity_id(
                effective_filters["run_id"], "run_id"
            )
        if not any(key in effective_filters for key in ("user_id", "agent_id", "run_id")):
            raise ValueError(
                "filters must contain at least one of: user_id, agent_id, run_id. "
                "Example: filters={'user_id': 'u1'}"
            )

        limit = top_k

        # Apply enhanced metadata filtering if advanced operators are detected
        if self._has_advanced_operators(effective_filters):
            processed_filters = self._process_metadata_filters(effective_filters)
            # Remove logical/operator keys that have been reprocessed
            for logical_key in ("AND", "OR", "NOT"):
                effective_filters.pop(logical_key, None)
            for fk in list(effective_filters.keys()):
                if fk not in ("AND", "OR", "NOT", "user_id", "agent_id", "run_id") and isinstance(effective_filters.get(fk), dict):
                    effective_filters.pop(fk, None)
            effective_filters.update(processed_filters)

        keys, encoded_ids = process_telemetry_filters(effective_filters)
        capture_event(
            "mem0.search",
            self,
            {
                "limit": limit,
                "version": self.api_version,
                "keys": keys,
                "encoded_ids": encoded_ids,
                "sync_type": "sync",
                "threshold": threshold,
                "advanced_filters": bool(filters and self._has_advanced_operators(filters)),
            },
        )

        original_memories = self._search_vector_store(query, effective_filters, limit, threshold)

        # Apply reranking if enabled and reranker is available
        if rerank and self.reranker and original_memories:
            try:
                reranked_memories = self.reranker.rerank(query, original_memories, limit)
                original_memories = reranked_memories
            except Exception as e:
                logger.warning(f"Reranking failed, using original results: {e}")

        return {"results": original_memories}

Four things in this signature matter more than the others. Retrieval is always scoped: the filter object must name a user, an agent or a run, and the method raises rather than searching the whole store when it does not — a memory lookup with no scope is a data-leak path, not a convenience. The score floor defaults to a low value, which is the right default only when the caller has another way to judge relevance; a host that pastes results straight into a prompt should raise it. rerank is opt-in, because a cross-encoder pass costs a second model call on every retrieval. And the metadata operators are declared in the docstring rather than hidden in an enum, so a caller can see that in, contains and the logical joins exist before writing a filter that silently matches nothing. The write side that fills this index is the other half of the same story.

Memory management on the write path: vectors plus payloads

A memory is not a string in a table. It is a vector plus a payload, written together, and the payload is what retrieval filters on later:

# backend/smartgate/core/resources.py — source lines 125–139 (store_vectors)
async def store_vectors(self, collection: str, vectors, payloads, ids=None) -> int:
        from qdrant_client.models import PointStruct
        import uuid
        points = []
        for i, (vec, pay) in enumerate(zip(vectors, payloads)):
            points.append(PointStruct(
                id=ids[i] if ids else str(uuid.uuid4()),
                vector=vec,
                payload=pay,
            ))
        result = await self._client.upsert(
            collection_name=collection,
            points=points,
        )
        return len(points)

The write path is deliberately one round trip: build the points, upsert them in a single call, return how many landed. Three details decide how the store behaves under load. Ids are optional — without one the code generates a UUID, which means a retry after a timeout creates a second copy rather than overwriting the first, so a writer that needs idempotency has to supply ids. Payloads travel next to the vectors in the same object, which is what lets the read path filter on scope without a second lookup. And the return value is a count, not a list: the caller learns that eight points were written, not which ones, so verification belongs to a follow-up query rather than to the write. Everything a memory system spends on ingestion is spent here — on the embedding that produced the vector and on the upsert that stores it.

Prompt compression before a write is paid for

The cheapest memory is the one you never store. Compression sits in front of the write path for the same reason it sits in front of a long prompt: most agent traces contain repetition, boilerplate and tool output that no later retrieval will score highly. Cutting them first changes two bills at once. The embedding is charged per token, so a trace that is compressed to half its length costs half as much to ingest. The vector store is charged per point, so paragraphs merged into one summary point cost one retrieve slot instead of five.

The pattern that survives contact with real traces is pre-filtering rather than summarization. A goal string — the step intent or the user query — lets the writer drop paragraphs that cannot answer anything, and the gateway's context tool exposes exactly that as a purpose argument next to a target ratio. What must not happen is silent loss: a compressed trace that cannot be re-expanded is a support ticket, so keep the raw body where a run can still be audited and store the compressed form as the retrievable one. Compression is a write-time decision with a read-time consequence.

Prompt injection protection: a memory store is where secrets accumulate

Anything an agent reads can end up in memory, and memory is replayed into future prompts. That makes the store a second place a secret can leak from, and it makes output masking a small but real control:

# backend/tools/cursor_mcp_deep_test.py — source lines 47–50 (mask_key)
def mask_key(key: str) -> str:
    if len(key) <= 12:
        return "***"
    return f"{key[:8]}...{key[-4:]}"

The function is three lines and the policy is worth stating: keys shorter than the visible window collapse entirely to asterisks, and longer ones keep a prefix and a suffix so an operator can still tell two credentials apart in a log. Masking at the boundary is a backstop, not a redaction system — it protects the places the gateway prints, not every place a value can travel. The stronger control is upstream: never write a credential into a memory payload, because a payload is exactly what the read path filters on and returns. Treat the store as public within its scope and the masking helper becomes what it should be — the last check before a key reaches a transcript.

Cost optimization: stopping the worker that keeps paying

Reads and writes are per-call costs. The costs that surprise people are the ones attached to a process, and a memory service usually owns at least one background worker:

# backend/smartgate/core/events.py — source lines 49–56 (stop)
async def stop(self):
        if self._worker:
            self._worker.cancel()
            try:
                await self._worker
            except asyncio.CancelledError:
                pass
            logger.info("EventBus stopped")

Five lines and one behaviour: if a worker exists, cancel it and await the cancellation, swallowing the CancelledError the task raises as it unwinds. That last part is the difference between a clean shutdown and a shutdown that logs a stack trace nobody will read. The reason this belongs on a page about memory cost is arithmetic rather than style. A worker holds a connection, polls a queue, or sweeps expired points; every version of that continues to bill compute while the deployment is idle, and a fleet of idle replicas multiplies it. Shutting the worker down on the same path that closes the server is how the cost per deployment stays proportional to the traffic it actually served.

Claude Desktop memory config: the file the host reads

A tool is only reachable where the host has been told to look. Each host reads its own configuration file in its own shape, and the desktop client's block is the one people copy first:

# lib/connect/mcp-config-templates.ts — source lines 76–96 (buildClaudeDesktopMcpConfigJson)
function buildClaudeDesktopMcpConfigJson(
  mcpUrl: string,
  apiKeyPlaceholder: string = API_KEY_PLACEHOLDER,
): string {
  return JSON.stringify(
    {
      mcpServers: {
        smartgate: {
          type: "streamable-http",
          url: mcpUrl,
          headers: buildMcpAuthHeaders(
            apiKeyPlaceholder,
            PLATFORM_AGENT_ID["Claude Desktop"],
          ),
        },
      },
    },
    null,
    2,
  );
}

The generated block is the whole contract in fifteen lines: a server name, the streamable-http type, the endpoint URL and an authorization header. Two properties make it worth generating instead of documenting. The key placeholder travels as a named argument with a default, so the block a user pastes and the block the server tests are produced by the same function. And the agent-platform header is attached per host, which is what lets an audit row say Claude Desktop rather than unknown client — the difference between a log that answers questions and a log that only records that something happened.

Cursor setup for the memory tool: an install command instead of JSON

Not every host wants a file edit. Two of them will take a command that writes the configuration themselves, which removes the most common setup failure — editing a file the host never reads:

# lib/connect/mcp-config-templates.ts — source lines 213–225 (getMcpPlatformInstallCommand)
function getMcpPlatformInstallCommand(
  platform: McpPlatform,
  mcpUrl: string,
): string | null {
  switch (platform) {
    case "OpenClaw":
      return buildOpenClawMcpSetCommand(mcpUrl);
    case "Claude Desktop":
      return buildClaudeDesktopMcpAddCommand(mcpUrl);
    default:
      return null;
  }
}

The switch is small on purpose. OpenClaw and Claude Code each get their own command builder; every other platform returns nothing, because the honest answer for the rest is edit the JSON and a generated command would be a guess. Returning a null instead of a plausible-looking command is the kind of decision that shows up in support volume rather than in a demo: a user who is told there is no installer goes to the file path, while a user handed a wrong command goes to the issue tracker.

The roots of the client config: one filename per host

Whether the host takes a command or a file, the file has a name, and the name is per-platform:

# lib/connect/mcp-config-templates.ts — source lines 249–264 (mcpConfigFilename)
function mcpConfigFilename(platform: McpPlatform): string {
  switch (platform) {
    case "Claude Desktop":
      return "claude_desktop_config.json";
    case "Cursor":
      return "mcp.json";
    case "Windsurf":
      return "mcp_config.json";
    case "Hermes":
      return "mcp.json";
    case "OpenClaw":
      return "openclaw-smartgate-snippet.json";
    default:
      return "smartgate-mcp.json";
  }
}

Six cases and a default. Claude Desktop reads claude_desktop_config.json; Cursor and Hermes both read mcp.json; Windsurf reads mcp_config.json; OpenClaw takes a namespaced snippet; anything else falls back to a generic name. This is the layer where "the server works but the agent cannot see it" is actually decided — a correctly built MCP server is invisible to a client whose configuration was written into the wrong file. Deriving the name from the platform keeps the docs, the installer and the audit label agreeing about which file a given host owns.

Config JSON a generic host can paste: the memory tool it enables

The config is only interesting because of what it reaches. Once a host is pointed at the endpoint, the tool surface is whatever the server registered — and for memory that is one tool with an action parameter rather than a family of endpoints:

# backend/smartgate/api/mcp.py — source lines 109–126 (smart_fetch)
@server.tool(
        name="smart_fetch",
        description=TOOL_DESCRIPTIONS["smart_fetch"],
        annotations=tool_annotations("smart_fetch"),
    )
    async def smart_fetch(
        url: str = Field(description="Full HTTP or HTTPS URL to fetch."),
        timeout: int = Field(default=30, description="HTTP timeout in seconds."),
    ) -> str:
        _, registry = _app_state()
        module = registry.get("fetch")
        ctx = _tool_ctx()
        return await _run_with_audit(
            "fetch",
            ctx,
            module.process(ctx, url=url, timeout=timeout),
            {"url": url},
        )

The declaration is the contract the host reads: the tool name, a human-readable description, and annotations derived from the tool's risk. The body then resolves the module from the registry, builds a call context, and returns through one audited exit — so a memory call and a fetch call produce the same shape of audit record. That is what makes the config block worth generating per platform: the URL is the same for every host, and the accounting is the same for every tool.

Cursor memory config: one platform id, one filename, same block

The last piece of wiring is the platform identifier, and it is the only thing that differs between two hosts using the same transport:

# lib/connect/mcp-config-templates.ts — source lines 56–73 (buildCursorMcpConfigJson)
function buildCursorMcpConfigJson(
  mcpUrl: string,
  apiKeyPlaceholder: string = API_KEY_PLACEHOLDER,
): string {
  return JSON.stringify(
    {
      mcpServers: {
        smartgate: {
          type: "streamable-http",
          url: mcpUrl,
          headers: buildMcpAuthHeaders(apiKeyPlaceholder, PLATFORM_AGENT_ID.Cursor),
        },
      },
    },
    null,
    2,
  );
}

This is the Claude Desktop block with one substitution: the platform id comes from a typed lookup keyed by the platform name. Everything else — the server map, the type, the URL, the header building, the two-space JSON formatting — is shared code. That is the argument for generating client configuration rather than publishing a snippet per host: a snippet is a copy that goes stale in the next release, while a lookup keyed by platform changes in one place. For a memory tool the stake is retrieval itself: if the header is wrong, the calls still look like they ran, and the memory they wrote lands under a different platform than the one the audit view groups by.

Testing memory in an mcp playground

Every test of a memory system is a retrieval test, and the only way to run one is against a live server. That is the job an MCP playground or inspector does: connect to the endpoint, call tools/list to see what the server advertises, then call the memory tool with an add and follow it with a search that should return what you just wrote. Two assertions are worth automating rather than eyeballing. The first is that a search with no scope fails instead of returning everything — the rejection is the feature. The second is that a memory written by one key is invisible to another, because tenant isolation lives in the filter, not in the vector index. The tooling question is in MCP inspector alternatives.

JSON-RPC underneath the memory tool

Nothing on this page is special to memory at the protocol level. The transport is JSON-RPC 2.0 over one Streamable HTTP endpoint: initialize, then notifications/initialized, then tools/list, then a tools/call whose arguments are the memory operation and its fields. That has a practical consequence for anyone debugging retrieval. A memory call that returns a JSON-RPC error object never reached the store, while a memory call that returns a result with an empty list reached the store and found nothing above the threshold — two different bugs with two different fixes. Read the response envelope before you read the vector store, and the wire format is worth knowing in detail: the message format is where the error shapes are spelled out.

---

How SmartGate compares

	What you store	When retrieval runs	What you pay
No memory	Nothing; every run starts from the prompt	Never	The prompt token bill, every turn
Transcript in a vector store	Raw turns, embedded as they arrive	On every lookup, with no scope or floor	Embeddings for the noise you also embed, plus a search per turn
Framework memory class in-process	Objects in the agent's own process	Until the process exits	Nothing extra, until a restart empties it and the run forgets
SmartGate memory as a tool — one of seven smart_* tools	Vector plus payload, scoped to user, agent or run	On action=search, with a threshold and a result cap	Metered like any other gateway call: 2M tokens a month on the free tier, 120 MCP requests a minute per key

The comparison is not about which store is faster. It is about where the three decisions live. If the memory class ships inside the agent, the scope rules, the score floor and the accounting are all in code the host can change; if it is a tool behind one endpoint, the same three decisions apply to every host at once. The middle column is the one people get wrong: retrieval that runs eagerly on every turn is the most common reason a memory-enabled agent costs more than the context window it was supposed to protect.

How to get started

1. Decide what is worth storing. Take one agent run and mark the passages a future run would score highly. Everything else is embedding cost.
2. Pick the scope. User, agent or run — the filter is mandatory, so this is the first decision, not a later hardening step.
3. Attach the tool to one host. The per-platform bindings and the endpoint contract are on the connect page and the MCP endpoint page.
4. Write, then search, then check the audit row. One add, one search, and one row in the logs view proving the call reached the gateway.
5. Read the catalogue before you extend it. The tool arguments, including the memory action set, are catalogued in the MCP tools reference.

Start on the free tier — 2M tokens a month, all seven tools, 120 MCP requests a minute per key — and compare the plans on the pricing page once the retrieval hit rate is something you can measure.

Limitations and what this does not do

• A threshold is not a guarantee of relevance. A high-scoring memory can still be wrong, stale or written by a different workflow; the score floor reduces the volume of bad recalls, it does not judge them.
• Scoping rules are only as strong as the caller. The read path requires a scope and the store filters on it; a host that passes a shared user id across tenants has still made every memory visible to everyone.
• Nothing here deduplicates across runs. Two runs that observed the same fact write two points unless the writer checks first, and the write path does not check for you.
• These are windows, not whole files. Each excerpt is a slice of a longer module, so the surrounding validation, telemetry and error handling are described rather than quoted.
• Compression is lossy by design. A trace that is compressed before it is stored is not recoverable from the store, so raw audit material has to be kept somewhere else.
• This is not a model host and it is not a retrieval-quality guarantee. It moves memory calls through one governed endpoint; it does not choose the embedding model for you.

Sources

• Model Context Protocol — specification (2026-07-28): https://modelcontextprotocol.io/specification/2026-07-28
• Model Context Protocol — transports, including stateless Streamable HTTP: https://modelcontextprotocol.io/specification/2026-07-28/basic/transports
• Model Context Protocol — server tools (tools/list, tools/call): https://modelcontextprotocol.io/specification/2025-06-18/server/tools
• Qdrant — points, ids and upsert semantics: https://qdrant.tech/documentation/concepts/points/
• Anthropic — introducing the Model Context Protocol: https://www.anthropic.com/news/model-context-protocol
• SmartGate — docs, connect, logs and pricing: https://smartgate.network/docs · https://smartgate.network/pricing

Method note

The code in this article is not transcribed. Each block was cut directly out of the slice body returned by the SmartGate slice API and re-asserted byte-for-byte as a substring of that body before publication; the first line inside every fence records the file and the exact source lines. Symbols were pinned by whole-name containment (rule A level 2) and confirmed by the service's slot-proof endpoint before any prose was written. All nine excerpts are whole slice bodies. Sections 2:1 (prompt compression, no-slice), 4:1 (mcp playground, abstain) and 12:1 (json-rpc mcp, abstain) pinned no slice and are written from the public specification and the project's own endpoint behaviour; no code is quoted for them, and the abstentions are recorded rather than hidden.

Demand figures come from this project's own keyword run, recorded in research_brief.md and search_volume.json — the section phrases prompt compression (140), mcp playground (70), prompt injection protection (50), context window management (40), claude desktop mcp config (40), cursor mcp setup (40), mcp roots (30), mcp config json (30), cursor mcp config (30), llm cost optimization (30), json-rpc mcp (20) and semantic dedup (10), each with its competition band. The head term agent memory and its 1,300 monthly searches were measured in the same research pass.

Slice provenance

#	SERP keyword	Symbol	File	Source lines	How it was pinned	sha256(12)
1	semantic dedup	Memory.search	backend/smartgate/modules/memory/algorithm.py	1126–1237	rule A L2 → slot-proof	aa50fd88cf6d
2	context window management	store_vectors	backend/smartgate/core/resources.py	125–139	rule A L2 → slot-proof	d44a629fc212
3	prompt injection protection	mask_key	backend/tools/cursor_mcp_deep_test.py	47–50	rule A L2 → slot-proof	1ac4b9165914
4	llm cost optimization	stop	backend/smartgate/core/events.py	49–56	rule A L2 → slot-proof	f4ddd0bc4e20
5	claude desktop mcp config	buildClaudeDesktopMcpConfigJson	lib/connect/mcp-config-templates.ts	76–96	rule A L2 → slot-proof	8f2a75a4ac53
6	cursor mcp setup	getMcpPlatformInstallCommand	lib/connect/mcp-config-templates.ts	213–225	rule A L2 → slot-proof	8ee797e82a2b
7	mcp roots	mcpConfigFilename	lib/connect/mcp-config-templates.ts	249–264	rule A L2 → slot-proof	d60503ebfb68
8	mcp config json	smart_fetch	backend/smartgate/api/mcp.py	109–126	rule A L2 → slot-proof	fdc9a4259783
9	cursor mcp config	buildCursorMcpConfigJson	lib/connect/mcp-config-templates.ts	56–73	rule A L2 → slot-proof	5ac3816f50b3

Every fenced block above was cut from the slice body and re-asserted against it byte-for-byte before publication. 9 of 12 sections pinned, 2 abstentions, 1 misses.