Token Optimization Techniques for AI Apps: Caps and Cost Control

Short answer: Token optimization for AI apps is four disciplines in one request path: measure tokens before you spend them, shape and compress the payload before the model sees it, cache and route so the same bytes are never paid for twice, and cap the loop with quotas and output limits. SmartGate implements all four in gateway code, and this page shows the mechanisms and the numbers behind them.

Key takeaways

• Measure with the model's own tokenizer. A count that comes back as an integer is something a quota can compare; a character estimate is not.
• Shape before you cap. Compression and deduplication reduce what the model reads, which is cheaper than refusing the request after it has been built.
• One URL per client. A hosted MCP endpoint replaces a private key per tool, and the host config is a short JSON file the gateway generates.
• Cache the repeats, not the decisions. A 45-second window on a budget snapshot is fine; a cached authorisation decision is not.
• Count what you saved. Two audit fields turn "we compressed something" into a number a finance team can read.
• Cap last, but cap. Per-minute rate limits, monthly token quotas and output limits are the backstop when shaping, caching and routing have not been enough.

---

The four levers: shape, cache, route, cap

Every technique on this page belongs to one of four levers, and they pay off in that order. Shaping is decided at the call site: fewer bytes leave your process. Caching removes a call entirely instead of making it cheaper. Routing changes which model answers, and at what price. Capping is the backstop that holds when the first three miss.

Most teams start with the cap, which is the most expensive place to begin. A monthly quota tells you the bill is too high; it does not tell you which of the three cheaper levers would have avoided it.

Lever	What it changes	Concrete control	Where it runs
Shape	the bytes that reach the model	compression, deduplication, compact tool output, structured data instead of prose	the tool layer
Cache	how often the same bytes are paid for twice	response cache with an explicit TTL; cached budget snapshots	the gateway edge
Route	which model or provider answers, and at what tier	dynamic routing, fallbacks, tiering by task class	the gateway policy
Cap	the ceiling per call, per key, per team	output limits, monthly token quotas, per-minute rate limits	the plan catalog and the budget guard

The four levers fail differently, which is why the order matters. Shaping changes what the model reads. Caching changes whether it reads at all. Routing changes the price of the call that remains. Capping decides what happens when the other three are not enough. An agent loop that appends its own transcript every turn defeats shaping and caching at the same time, so measurement comes first: you cannot tell which lever is missing until you can see the number.

Why LLM token efficiency is a gateway metric

Token efficiency looks like a prompt-writing habit and behaves like a traffic problem. The cost of one agent turn is not the user's question; it is the question plus every tool definition the host sent, every result the loop accumulated, and every retry. A 2026 measurement of an identical ten-turn agent conversation counted 2,400 tokens per turn with no MCP servers attached, 18,700 with three, and 31,200 with five (MCP's dirty secret). A separate Q1 2026 dataset of 22 teams put MCP input tokens at 41–58% of total coding-agent spend (MCP gateway economics).

Nothing in the prompt changed between those runs. What changed was the tool surface, which is why the count has to be taken where the traffic is, not where the prose is written. And the count itself has a subtlety that decides whether anyone keeps taking it.

get_token_length is that measurement path: text goes in, an integer comes out, and the caller decides what to do with it. Two details make it worth reading. First, two tokenizer families sit behind one call and the function picks the matching one, so a team that switches host models does not have to switch its budget arithmetic. Second, it calls tokenizer.tokenize and adds the special-token count separately instead of calling the tokenizer's own __call__, because the latter warns and does more work on inputs longer than the model's maximum length — which is exactly the shape of input a fetch tool produces. A measurement that costs a warning on every large document is a measurement nobody runs on the hot path, and an unmeasured path is the one that produces the surprise invoice.

# backend/smartgate/modules/context_gate/algorithm.py — source lines 987–1003 (token count)
def get_token_length(
        self,
        text: str,
        add_special_tokens: bool = True,
        use_oai_tokenizer: bool = False,
    ):
        if use_oai_tokenizer:
            return len(self.oai_tokenizer.encode(text))
        else:
            # tokenize + special token count avoids tokenizer.__call__ on megabyte
            # texts (transformers warns when len > model_max_length).
            n = len(self.tokenizer.tokenize(text))
            if add_special_tokens and hasattr(
                self.tokenizer, "num_special_tokens_to_add"
            ):
                n += self.tokenizer.num_special_tokens_to_add(pair=False)
            return n

The special tokens are the detail teams drop first, and dropping them is worth a constant error per request: invisible once, significant over a month of agent traffic. Keeping the count separate from the decision is the other half of the design. The function returns an integer; the quota check, the dashboard and the audit row all consume the same number later, so measurement never has to be reimplemented per caller.

Those per-call numbers are also the input to a monitoring story, and turning them into traces, latency and failure views is what an LLM observability layer does with the same records. The LLM observability tools worth shortlisting are the ones that read the token count as a first-class signal rather than a footnote on a latency chart.

Windsurf MCP: one URL where a key per tool used to be

The practical objection to a gateway is configuration sprawl: every host wants its own file, its own key and its own transport setting. The fix is to make the host config a generated artifact rather than a document each developer transcribes by hand, and to point every host at one endpoint.

The excerpt below is the builder for one host, and it is short on purpose. The entry type is streamable-http rather than a command, so nothing is spawned locally; serverUrl is the single MCP URL; and the authorisation header is written from a placeholder, not from a live key, so a config committed by accident does not carry a credential. The platform constant in the last line is not decoration: it is what lets the request be attributed to the host that sent it when the activity log is read back.

# lib/connect/mcp-config-templates.ts — source lines 99–116 (host config)
function buildWindsurfMcpConfigJson(
  mcpUrl: string,
  apiKeyPlaceholder: string = API_KEY_PLACEHOLDER,
): string {
  return JSON.stringify(
    {
      mcpServers: {
        smartgate: {
          type: "streamable-http",
          serverUrl: mcpUrl,
          headers: buildMcpAuthHeaders(apiKeyPlaceholder, PLATFORM_AGENT_ID.Windsurf),
        },
      },
    },
    null,
    2,
  );
}

Two consequences for the token bill. First, the transport decision stops being a per-developer choice: if the host speaks Streamable HTTP, the same endpoint serves Cursor, Claude Desktop, Windsurf and a CI job, so a compression or caching change reaches every client at once instead of landing in one editor. Second, the tool surface stays in one place, which is the only way to keep it small. Five servers configured by hand add five tool lists to every request; one endpoint can expose a bounded set and route the rest server-side. The gateway generates the snippet for the host you connect, and the connect page lists the current hosts.

A CI job that drives that same endpoint is an automation like any other, and the retry, deduplication and receipt discipline it needs is described in AI automation for SaaS operations.

Prompt compression: shrink the payload before it is sent

Compression is the lever with the best ratio of effort to tokens, and it is also the one that breaks quality most quietly. It splits into two families.

Mechanical compression removes bytes that carried no information: duplicated passages across retrieved documents, navigation chrome around the paragraph you wanted, repeated whitespace and boilerplate, and tool output serialised as prose when a table would do. This family is lossless in practice and safe to enable by default, which is why it should be the first thing a team turns on.

Learned compression rewrites the prompt itself. LLMLingua's answer is a budget controller that sets how much may be cut and a token-level compression pass that keeps the remaining tokens interdependent, so a compressed prompt is not a bag of keywords (LLMLingua, paper). The budget controller is the part most implementations skip, and skipping it is how compression becomes a silent quality regression: without a stated budget, the compressor cuts until it fits instead of until the answer survives.

Three rules keep compression honest. Keep the instruction block out of the compressed region; compress retrieved evidence, not the question. Log the ratio per call rather than per week, so a regression is attributable. And decide the loss budget per task class, because a summarisation task tolerates more cutting than a code edit. The code-sensitive parts of the same problem — which tokens survive when context is tight — are in context window management techniques.

Caching: stop paying twice for the same bytes

An agent stack wants three different caches, and conflating them is how caching gets a bad reputation.

Response caching stores a completed model answer under a key built from the exact prompt, model and parameters. It is only correct when the key is exact: a cache that answers a "similar" prompt is a correctness bug with a good hit rate.

Tool-result caching stores what a fetch or retrieval returned for a URL or query inside a short window. This is where the largest savings usually sit, because agent loops re-read pages they already read three steps earlier.

Decision caching stores the answer to a cheap, frequently asked question, such as whether a team is inside its budget. SmartGate caches that snapshot for 45 seconds by default: the TTL is an environment override, and a value that is non-finite or not positive falls back to 45 rather than becoming an infinite window. The trade-off is explicit — a burst of tool calls shares one budget check, and the cap can be up to 45 seconds stale. That is the right trade for an allowance check and the wrong one for an authorisation decision, which is why tool-call authorisation is evaluated per request instead.

Two disciplines make the difference between a cache and a leak. Every entry needs a TTL, not just an eviction policy, because a stale entry that never expires is indistinguishable from a wrong one. And every cached response that reaches a user needs a key that includes the tenant: a cache shared across teams will eventually serve one team's data to another.

Where each of those caches belongs, and what it is worth on a date axis against the spend it offsets, is the subject of Build a Low-Cost AI Backend Architecture.

Cloudflare AI gateway: caching, routing, fallbacks

Cloudflare AI Gateway is a hosted control point in front of model providers, and its feature list reads like the four levers: response caching, rate limiting, guardrails, dynamic routing with fallbacks, per-request logging, and token and cost analytics (AI Gateway). Dynamic routing is the routing lever made concrete: rules combine provider, model, quota and condition, and a request that fails or exceeds its condition falls through to the next target rather than failing the turn (dynamic routing).

Read the scope carefully before assuming it solves a tool-traffic problem. An AI gateway governs model calls: the completion your app asks for. Most of the token amplification measured above happens one layer earlier, in the tool calls that fill the context before the completion is requested, and those are governed by an MCP gateway instead. The two layers compose: route and cache the completion, and compress, authorize and meter the tools that feed it. If the distinction is new, the AI gateway versus API gateway is the shorter read.

MCP server on AWS: where the loop runs and who pays

Hosting an MCP server on AWS is a solved problem with several answers: a container behind a load balancer, a Lambda function for infrequent tool calls, or a managed gateway. Amazon Bedrock AgentCore Gateway is the managed version of that last shape — one secure entry point for agentic traffic that routes to tools, to other agents and to models, with MCP servers declared as gateway targets and authentication, policy enforcement and observability consolidated at the endpoint (gateway targets, gateway overview).

The token-relevant point is what hosting does not change. Where the server runs affects latency, cold starts, egress and your operational surface, but the tokens a call consumes are decided by the payload the server returns and by how often the host re-sends its tool definitions. A server moved from a laptop to a managed gateway produces the same context, unless the move also puts compression, caching and metering in the request path. So choose the runtime on deployment grounds, and make the accounting a property of the request path rather than of the host. The wider deployment question — what the gateway layer should own versus what the application keeps — is covered in enterprise AI gateway architecture.

MCP server testing: assert the token counts, not just the answers

Tool tests usually assert that the right content came back. For a token-cost problem that is half a test suite, because a server can return the correct document in the most expensive possible shape. The assertions worth writing are the ones a cost regression would fail:

• Long input, stable count. A very large text should produce a token count within a small tolerance of the reference, so a tokenizer swap or a truncation bug shows up as a test failure instead of a bill.
• Chunking by tokens, not characters. A splitter that measures characters fragments differently across languages and markdown, and the fragments are what get compressed.
• Explicit degradation. An unknown model name should degrade in a defined way rather than throwing on the hot path; the accounting for it should be visibly zero rather than missing.
• Degenerate input. A single-token input and an empty result are the inputs that break ratio arithmetic.

The savings fields are the ones that turn compression from a claim into a number, and the excerpt below shows how they are attached. The function reads the raw and compressed token counts out of a tool result and adds them to the audit parameters — but only when the raw count is above zero, and it returns the parameters untouched when the result is not the dictionary shape it expects. That guard is the whole point: a failed compression must not be able to write a phantom saving, because the savings aggregate is what a bill is computed from.

# backend/smartgate/core/audit_params.py — source lines 4–15 (savings fields)
def enrich_compress_audit_params(
    params: dict,
    result_data: object,
) -> dict:
    """Add raw_tokens / compressed_tokens for savings aggregation (spec §4.1)."""
    if not isinstance(result_data, dict):
        return params
    origin = int(result_data.get("origin_tokens") or 0)
    compressed = int(result_data.get("compressed_tokens") or 0)
    if origin > 0:
        params = {**params, "raw_tokens": origin, "compressed_tokens": compressed}
    return params

The same pattern generalises to any metered tool: keep the raw count, keep the count after processing, and attach both to the audit record for the call. Two integers per call is what makes a monthly report reconcilable, and it is what lets a compression change be evaluated on evidence instead of on a feeling about response quality. That same audit record is what an AI agent observability setup reads, turning the two counts into a trace span and a spend line.

Two integers per call are also the smallest thing a finance reader can check, and the same numbers become quotas, alerts and a per-tool spend report on the FinOps lead page.

MCP stdio versus remote transports

The Model Context Protocol defines exactly two standard transports: stdio, where the client launches the server as a subprocess and exchanges JSON-RPC messages over standard input and output, and Streamable HTTP for remote servers. The specification's guidance is that clients should support stdio whenever possible (transports), and for a developer tool that is the right default: no port, no certificate, no auth handshake.

Token-wise the choice is narrower than it looks. A stdio server still declares its tools to the host, and the host still serialises those definitions into the context of every turn — the amplification measured in the first section happens with local servers too. What stdio changes is operations, not payload: the process is yours, the data stays on the machine, and there is nothing to authorise. What a remote endpoint changes is the opposite, and that is where the levers live: a hosted endpoint can compress a response before it returns, cache a fetch, and charge the call to a team. The practical rule is to pick the transport for the deployment you actually have, then make sure the token controls sit in the layer both of them call. The revision history matters here, because the remote transport changed once already.

RAG vs agentic RAG: what a loop adds to the bill

Classic retrieval-augmented generation retrieves once and sends the top passages with the question: one retrieval, one completion, a predictable payload. Agentic RAG lets the model choose its retrievals, so the number of hops becomes a runtime decision — and every hop re-sends the context accumulated so far, including the tool output of the previous hop (agentic RAG survey).

The bill therefore scales with hops rather than with documents. Two retrievals on one page cost more than one retrieval across two pages, because the second call carries the first result in its context. This is where the four levers stop being independent: compressing a retrieved passage pays twice, once when the passage is fetched and again on every later hop that includes it. The classic-versus-agentic trade-off and where to stop looping are covered in RAG versus agentic RAG; the compression side is above.

For budgeting, the useful move is to treat a hop as a metered unit. Give the loop a hop ceiling and a per-hop token allowance, then compare the two after the fact. A loop that hits its hop ceiling before its token allowance is reasoning; a loop that hits the token allowance first is re-reading.

Output limits: quotas, rate limits and per-call caps

Capping has three levels, and a complete setup uses all of them. Per-call output limits bound the most expensive surprise, because generated tokens are billed to your account and a runaway generation is invisible until it is finished. Per-minute rate limits bound the burst: they decide how fast a loop can spend. Monthly quotas bound the month and are the number a team actually plans against.

SmartGate's catalog makes the levels explicit rather than hiding them in a single number. The monthly token limits step 2M, 20M, 100M and 200M across the plans, and the rates step with them:

Plan	Monthly token limit	Team requests/min	MCP requests/min per key	Team ceiling
Free	2M	60	120	120
Pro	20M	120	300	600
Teams	100M	300	600	3000
Enterprise	200M	600	1200	9999

Read as a budget, the interesting column is the last two: a per-key limit is what stops one misconfigured client, and the team ceiling is what stops ten correct clients from arriving at once. How many of those clients may be in flight at the very same moment — the ceiling's other half — is the subject of concurrency control in AI backends. The implementation details — how a quota is enforced per team, what happens at the boundary, and how the decision reaches the tool call — are in enforcing a token quota per team.

How SmartGate compares

The gateway category splits by what sits in the request path. SmartGate is narrower than "AI gateway": it governs tool traffic, and its billing participates only after it has saved you something.

	What it governs	How you run it	What you pay
SmartGate	MCP tool traffic: fetch, compression, dedup, budgets, pipelines	Hosted gateway; one MCP URL per client	Free: 2M tokens/month and all seven tools. Pro from $18/month, first month $5; the share starts after $15 of measured savings and is capped at $18/month, so the total stays under $36/month (pricing)
Cloudflare AI Gateway	Model traffic: caching, rate limiting, guardrails, dynamic routing with fallbacks	Hosted gateway in front of model providers	Usage-based, from your provider relationship (AI Gateway)
Amazon Bedrock AgentCore Gateway	Agentic traffic to tools, agents and models, with MCP servers as targets	Managed AWS gateway	Priced with the rest of the service you deploy (overview)
Local compression servers (for example mcp-context-guard)	Context compression inside one developer's editor	A stdio server you install and run	Open source; you own the runtime (PyPI)

Two honest readings of that table. If your only problem is that one long document keeps filling an editor's context, a local compression server is a reasonable start: it is free and it does not touch your network. If the problem is enforcement — who may call what, how much, under which budget, with what audit trail — compression alone does not answer it, and a gateway does. The last line of the table is also the one to read twice: a locally installed server is convenient and unmetered, which is another way of saying nobody can tell you what it saved.

The seven tools themselves are documented in the MCP tools reference, and the per-request accounting that makes the savings share measurable is in the execution-cost breakdown.

How to get started

1. Connect one client. Generate the config for Cursor, Claude Desktop, Windsurf or your own host, paste it in, and confirm the seven tools appear in the tool list. This is the step that replaces a key per tool with one URL.
2. Turn on shaping before capping. Run a fetch, a retrieval and a dedup over a workflow you already own — competitor pages, documentation, issue threads — and watch the compressed against raw token counts on the usage report. If the ratio does not move, the payload was already small and the cap is not your problem.
3. Then enforce. Wrap the loop with a budget check and a record call, set the team's monthly cap, and give long research-shaped work a pipeline call so four steps become one audited request instead of four.

Start on the free plan — 2M tokens/month, all seven tools, no card — with start free, then check the caps against your own workload on the pricing page.

Limitations and what this does not do

• Compression is lossy. Splitting and compressing a long payload will drop content you wanted. Test the ratio on your own corpus before you rely on it, and keep the instruction block out of the compressed region.
• The decision cache trades freshness for cost. A 45-second window means the allowance reading can be up to 45 seconds behind. That is a deliberate trade for an allowance and the wrong one for an authorisation.
• The numbers here are measurements, not guarantees. The per-turn amplification figures come from a specific agent, a specific host and a specific set of servers; your tool surface decides yours.
• A gateway is not a sandbox. Governing which tools may be called does not make an unsafe tool safe. Pair it with least-privilege keys, role checks and a review of what your agent is allowed to reach.

Sources

• Model Context Protocol specification, Transports (stdio and Streamable HTTP): https://modelcontextprotocol.io/specification/2025-06-18/basic/transports
• Cloudflare AI Gateway, feature overview (caching, rate limiting, guardrails, dynamic routing): https://developers.cloudflare.com/ai-gateway/
• Cloudflare AI Gateway, dynamic routing and fallbacks: https://developers.cloudflare.com/ai-gateway/features/dynamic-routing/
• Amazon Bedrock AgentCore Gateway, MCP servers as targets: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway-target-MCPservers.html
• Amazon Bedrock AgentCore Gateway overview: https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/gateway.html
• LLMLingua: prompt compression with a budget controller: https://llmlingua.com/ · https://arxiv.org/abs/2310.05736
• Agentic Retrieval-Augmented Generation survey: https://arxiv.org/abs/2501.09136
• Per-turn token amplification in an agent conversation (2,400 / 18,700 / 31,200): https://dev.to/mrclaw207/mcps-dirty-secret-your-agent-is-burning-10-32x-more-tokens-than-you-think-32d1
• MCP input tokens as a share of coding-agent spend (22 teams, Q1 2026): https://futureagi.com/blog/how-mcp-gateway-cuts-token-costs-claude-code-codex-cli-2026
• MCP ecosystem statistics (SDK downloads, public server counts): https://agentscamp.com/guides/mcp/mcp-ecosystem-statistics
• SmartGate plans and pricing: https://smartgate.network/pricing

Method note

The code excerpts in this article are not transcribed. Each block was cut directly out of the slice body returned by the SmartGate slice API and re-asserted byte-for-byte as a substring of that body before publication; the first line inside every fence records the file and the exact source lines it came from. Symbols were pinned by whole-name containment (rule A level 2) and confirmed by a server-side proof call before any of them entered the text.

Three of the eight planned sections resolved to a symbol that could be pinned, and those are the three excerpts below. The remaining five are written from the protocol specification, the provider documentation and the product's own plan catalog, which is the house rule: a section that cannot be pinned is sourced, never invented. Repository-relative paths are shown as they are in the source; internal hosts, credentials and private addresses were stripped before any of it reached this document.

Slice provenance

#	SERP keyword	Symbol	File	Source lines	How it was pinned	sha256(12)
1	llm token efficiency	PromptCompressor.get_token_length	backend/smartgate/modules/context_gate/algorithm.py	987–1003	rule A L0 → remote symbol candidates → slot-proof	1ed8b361a397
2	windsurf mcp	buildWindsurfMcpConfigJson	lib/connect/mcp-config-templates.ts	99–116	rule A L3 → slot-proof	82aabcf99dd1
3	mcp server testing	enrich_compress_audit_params	backend/smartgate/core/audit_params.py	4–15	rule A L0 → remote symbol candidates → slot-proof	952380769807

Every fenced block above was cut from the slice body and re-asserted against it byte-for-byte before publication. 3 of 8 sections pinned, 0 abstentions, 5 misses.