SmartGateSmartGate

AI Gateway vs API Gateway: Where Each Responsibility Sits

An AI gateway and an API gateway split responsibilities rather than generations. Both route a request, check the caller's identity and apply a request rate limit. They part company at token quota per key and team, the cost record per call, the audit row a task is closed with, and the non-model tools an agent can reach.

Short answer: An AI gateway and an API gateway split responsibilities rather than generations. Both route a request, check the caller's identity and apply a request rate limit. They part company at token quota per key and team, the cost record per call, the audit row a task is closed with, and the non-model tools an agent can reach. A classic gateway forwards bytes it never reads; the AI half has to price and attribute what those bytes contain.

Key takeaways

  • Routing, auth and rate limiting are shared ground. Three of the rows below are drawn the same way in both products; only the key a decision is counted against changes.
  • The split starts at quota and accounting. Requests and bytes are counts a REST gateway can take; a day-scoped token counter per team and a measured record per task are written by a different layer.
  • The record is the product. A task row with a correlation id, a span count, a token total and the tools a loop called is what makes an overrun explainable three weeks later.
  • Paths and refusals are contracts. A canonical URL derived from a kind and a slug lets a machine consumer re-fetch an artifact without a route table, and a typed cap error tells a client when to come back instead of inviting a blind retry.
  • Do this next: write the responsibility table below for your own stack, name one owner per row, then decide which half is missing rather than which product is fashionable this quarter.

The two-minute version, for whoever signs off on the gateway

A gateway that terminates REST traffic already answers three questions: where does this request go, is the caller allowed in, and how many requests per minute do we accept. Those are transport and identity concerns, and they are mature.

The AI side adds four questions a REST gateway has no vocabulary for. How many tokens this team may spend before the day ends. What one task cost across a model call, a fetch and a compression step. Who called, and how the loop can be reconstructed afterwards. And which non-model tools the caller can reach, because those are what make an agent loop expensive rather than merely busy.

SmartGate is one implementation of that side - an MCP-native algorithm gateway for token control, traffic shaping and agent audit - and it does the three shared things as well. If you already run an API gateway, most of the AI side can be built behind it; accounting is the hard half, because it needs the request context that only the layer nearest the payload holds.

What the search results for ai gateway vs api gateway actually show

Three measured facts frame this page. Google Ads reports about 20 US searches a month for this comparison phrase at LOW competition (index 7), measured on 2026-09-21, after a 70-search August; the neighbouring phrase api gateway llm measures 10 a month at MEDIUM competition; and the six mechanism phrases this page covers in code - agent workflow memory, anthropic model context protocol, mcp authorization, model context protocol docs, rag vs agentic ai and token optimization - each measure 170 a month. That mix is why the page spends its depth on mechanisms rather than on definitions.

The live SERP is small - 134 results - but carries an AI Overview, four People Also Ask questions ("What is an AI API gateway?", "Do I need an AI gateway?", "Is nginx considered an API gateway?", "Does AWS have an AI gateway?") and eight related searches, among them "Kong AI Gateway" and "Ai gateway vs api gateway vs aws". Vendors hold the top three organic results: Kong, Traefik and TrueFoundry. Two caveats: the AI Overview was present but its text and references came back empty, so this page claims only that it exists; and no difficulty score is quoted, because the endpoint returned no row for the phrase.

Vendor documentation is the source of record, and it disagrees usefully about scope. Google Cloud describes API Gateway as secure access to backend services through a well-defined REST API; Cloudflare describes AI Gateway in its own words as a way to "observe and control your AI applications with analytics, caching, rate limiting, and model fallback"; Azure API Management names "LLM deployments, AI APIs, and MCP servers accessed by your AI apps and agents"; Traefik says an AI gateway "governs LLM traffic with rate limits, PII redaction, and observability"; Portkey sells "a unified interface for interacting with over 250 AI models"; and LiteLLM and Envoy AI Gateway come at the same ground from the proxy side. The vendors agree about model traffic and disagree about everything around it.

The responsibility map, before any code

Each row is one responsibility; the last column names the excerpt later in this page that shows it in shipped code.

Responsibility Classic API gateway AI / algorithm gateway Shown by
Request routing Path, host and version tables The same, plus a core that picks the model or tool urlPrefix
Authentication Session or bearer token validation The same, plus a key that resolves to a team kindFromPublishPath
Request rate limiting Per-IP or per-route windows The same window, keyed by key or team dailyCapKey at day scale
Token quota per key or team Not modelled A day-scoped counter per team, refused with a typed error PlaygroundDailyCapError, dailyCapKey
Cost accounting Not modelled Token totals, span counts and tool lists per task transformAuditTask
Audit and trace Access logs One row per task, with a correlation id and a status transformAuditTask, normalizeTaskStatus
Artifact path contract A route file, edited per endpoint A path derived from kind and slug, read back the same way urlPrefix, publishPathFromKindSlug, kindFromPublishPath
Tool / MCP proxying Not modelled Tools reached through the same identity and the same record the hydrated upstream below
Error shape A fixed envelope: 429, 403 Typed refusals that carry a retry hint PlaygroundDailyCapError

Argue about the fourth row in review, because that is where the money is: everything above it can be assembled from an API gateway plus middleware, while quota and accounting are where custom code starts costing more than a dedicated layer.

Deployment shapes: where the two layers can physically sit

Shape What sits in the request path What stays yours to build
One gateway, both halves Routing, auth and rate limiting at the edge; the same product meters tokens and writes the task row Nothing, if the product really parses payloads - ask for the token record, not the access log
API gateway, AI layer behind it The classic gateway routes a prefix to a core; the core resolves the key, meters and audits The core, and the rule that only the core may reach a model
A model-proxy AI gateway beside the classic one Model calls get their own gateway; REST and tool traffic keep the old one Quota for non-model calls, and one record that covers both surfaces
A gateway in front of a publishing core The upstream assembles artifacts that humans and agents both fetch by path The path contract, and a cache that never serves a half-hydrated page

The rule that decides whether a shape works is where the payload is parsed: a byte-level proxy can forward a request it cannot describe, so accounting in the wrong layer produces a meter that measures the wrong thing. The same four shapes, with ownership marked per box, are drawn in the enterprise reference architecture, and the private-cloud deployment is the variant where the upstream address is configuration rather than a literal.

What the gateway's upstream serves: blocks hydrated before a reader sees them

Start behind the gateway rather than in front of it, because that is what a request ultimately returns. The function below is the hydration step of a publishing core: stored blocks arrive as untyped records and every text block is turned into HTML before anything renders.

# lib/pseo/hydrate.ts — source lines 6–27 (hydratePseoBlocks)
async function hydratePseoBlocks(blocks: unknown): Promise<PseoBlock[]> {
  if (!Array.isArray(blocks)) return [];

  const out: PseoBlock[] = [];

  for (const raw of blocks) {
    if (!raw || typeof raw !== "object") continue;
    const b = raw as PseoBlock;
    if (b.type === "text") {
      const tb = b as TextBlockData;
      const fmt = tb.contentFormat ?? "markdown";
      const html = await markdownToSafeHtml(tb.content, {
        asRawHtml: fmt === "html",
      });
      out.push({ ...tb, content: html, contentFormat: "html" });
    } else {
      out.push(b);
    }
  }

  return out;
}

Three decisions there are the shape of this kind of upstream. The guard refuses a value that is not an array instead of throwing, so a missing artifact renders as an empty page rather than a 500. The block's own declared format decides whether its content is read as markdown or passed through as raw HTML. And the conversion is asynchronous, because markdown-to-HTML is the slowest thing in an otherwise pure read path - and caching is what a read path buys you. A gateway forwarding this response sees a status code and a byte count, which is exactly the information that cannot tell you what the request cost.

Gateway path shapes: what a URL prefix decides before a request is parsed

An API gateway routes on the request line - host, path, method - and the mapping from a path family to a backend lives in configuration, so adding a family means editing that configuration. The function below makes the same decision in data instead: the public prefix for a kind comes from a lookup, and a kind the map does not know falls back to itself.

# lib/pseo/kind-routing.ts — source lines 36–38 (urlPrefix)
function urlPrefix(kind: string): string {
  return KIND_TO_URL_PREFIX[kind as ConsumerKind] ?? kind;
}

The fallback is the interesting part. An unknown kind is not an error at this layer; it becomes its own prefix, so a new artifact family can appear at a URL without a deployment, and a mistyped kind produces a new public path family rather than a refusal. That is the trade a catch-all route makes, and it is defensible only when something downstream validates. Keep a classic gateway in front of this core and the prefix map is what its route table has to agree with - two copies of one truth.

From kind to published path: the canonical URL an agent workflow can re-fetch

A machine consumer that holds a stored identifier should be able to build the URL for it. The function below is that rule in its smallest form: given a kind and a slug, return the path.

# lib/pseo/kind-routing.ts — source lines 45–47 (publishPathFromKindSlug)
function publishPathFromKindSlug(kind: string, slug: string): string {
  return pseoUrlPath(kind, slug);
}

The path is a pure function of two strings, so nothing has to be registered for the URL to exist, and an agent workflow that remembers a kind and a slug can re-fetch the artifact without a lookup. An API gateway's route table cannot make that promise: it is an operational artifact, and a route that is not in it does not exist. This is the first place the two layers differ in capability rather than in degree, and the difference is who owns the naming. The mirror of it is agent memory storage: once a loop remembers a reference, that reference has to stay resolvable, or the memory becomes a list of dead ends.

Reading the path back: anthropic MCP clients and the kind lookup

The inverse direction is where classification happens. The function below takes a public URL, splits it, and returns the kind and slug it encodes - or nothing at all.

# lib/pseo/kind-routing.ts — source lines 66–81 (kindFromPublishPath)
function kindFromPublishPath(urlPath: string): ParsedPublishPath | null {
  const path = normalizePath(urlPath);
  const parts = path.split("/").filter(Boolean);

  if (parts.length !== 2) {
    return null;
  }

  const [prefix, slug] = parts;
  const kind = URL_PREFIX_TO_KIND[prefix!];
  if (!kind || !slug) {
    return null;
  }

  return { kind, slug };
}

Read the strictness as the design: exactly two path segments, both non-empty, and a prefix the map knows. Anything else returns null rather than a default, so a caller cannot publish under the wrong prefix because a segment was misspelled. The direction is load-bearing for clients, not only for servers: a client that receives a URL should be able to say what it is looking at, and Anthropic MCP naming documents how much of that vocabulary is still settling. An API gateway classifies traffic by path family and stops there; a gateway whose upstream refuses to guess does not create publishable garbage.

mcp authorization, recorded: how a task outcome becomes a status

Authorization is a decision, and a decision that is not recorded is not auditable. The function below is the seam between a database column and a status enum: three literal values pass, everything else becomes an error.

# lib/smartgate/audit-logs.ts — source lines 180–185 (normalizeTaskStatus)
function normalizeTaskStatus(raw: string): TaskStatus {
  if (raw === "success" || raw === "error" || raw === "partial_error") {
    return raw;
  }
  return "error";
}

The whitelist is conservative in a specific direction. An unrecognised status reads as a failure - the safe default for a compliance surface and the wrong default for a dashboard, because a new legitimate outcome added to the writer without the reader is counted as an error until somebody updates the list. That trade belongs in review next to the authorization model itself: scoped tool credentials covers who may call what, and this function covers what the record says happened afterwards. The two are usually owned by different teams, which is how a system ends up with strong mcp authorization and unusable audit rows.

The row behind model context protocol docs: correlation id, tools, token totals

If the status is the outcome, this is the record. The function below maps one stored task row into a typed summary: a correlation id, a start and an end time, a span count, a token total, the normalised status and the tools the run called.

# lib/smartgate/audit-logs.ts — source lines 187–198 (transformAuditTask)
function transformAuditTask(raw: Record<string, unknown>): AuditTaskSummary {
  const tools = raw.tools;
  return {
    correlationId: String(raw.correlation_id ?? ""),
    startedAt: String(raw.started_at ?? ""),
    endedAt: String(raw.ended_at ?? ""),
    spanCount: Number(raw.span_count ?? 0),
    totalTokens: Number(raw.total_tokens ?? 0),
    status: normalizeTaskStatus(String(raw.status ?? "error")),
    tools: Array.isArray(tools) ? tools.map(String) : [],
  };
}

That is the accounting shape a REST gateway has no place for. An access log line records a method, a path, a status and a duration - the fields the request line and the clock provide - and none of them can answer what the call cost. The fields here come from inside the payload, spans and tokens, so they can only be written by the layer that parsed it, which is why what an agent run costs is a different job from enabling gateway logging. Publish the row where audit logging and retention can be reasoned about together, and treat the zeros as unknown rather than free: a missing count becomes zero here, and a zero that means "not measured" eventually reads as a saving. The same row is what a documentation consumer asks about, which is why its structure matters as much as the prose in the MCP documentation set.

rag vs agentic ai at the ceiling: the refusal a classic gateway cannot express

A cap is only a cap if its refusal is distinguishable from every other failure. The class below is a typed error carrying a machine-readable code and a retry hint.

# lib/smartgate/playground-daily-cap.ts — source lines 5–14 (PlaygroundDailyCapError)
class PlaygroundDailyCapError extends Error {
  readonly code = "PLAYGROUND_DAILY_CAP" as const;
  readonly retryAfterSeconds: number;

  constructor(retryAfterSeconds: number) {
    super("Playground daily request limit reached for this team.");
    this.name = "PlaygroundDailyCapError";
    this.retryAfterSeconds = retryAfterSeconds;
  }
}

The error is catchable by type, so a caller can answer "wait and come back" instead of "your request was malformed", and the retry hint travels with the error rather than being inferred from a header. The message names a team, because the ceiling is per team. The distinction between the two halves of an AI stack is what makes the shape necessary: a single retrieval call is cheap, repeatable and read-only, while a tool-calling loop compounds - the line the reading-versus-acting split draws on the metrics side shows up here as a budget side. what an MCP gateway adds is where the refusal gets attached to a tool surface rather than to one product's playground.

Token optimization at the quota layer: one counter per team per day

The counter key is the quota. The function below builds it from a team id and a date, in one line.

# lib/smartgate/playground-daily-cap.ts — source lines 16–19 (dailyCapKey)
function dailyCapKey(teamId: string, date = new Date()): string {
  const day = date.toISOString().slice(0, 10);
  return `playground_daily:${teamId}:${day}`;
}

Three properties follow from that string. The window is a calendar day in UTC, so a team working past midnight sees the counter reset on someone else's clock - acceptable for a fair-use ceiling, wrong for an invoice. The counter is keyed by team rather than by key, so one leaked key can spend a whole team's day; per-key granularity is the next dimension and costs one more field in this function. And the key is derived rather than stored, so the reset is free. Token optimization at this layer is not about compressing prompts - it is about which counter a refusal can be attributed to, and per-team token quotas are the subject of a longer argument.

How SmartGate compares

Each alternative is described the way its own documentation does; the links are the source.

What it actually governs Where the policy lives What you pay
Build it per service Whatever each service remembers to check; quota and cost assembled afterwards Scattered across services and repositories Engineer time, plus the incident nobody can reconstruct
A classic API gateway (Google Cloud, AWS, NGINX) Routing, authorization, request rate limits, access logs In its configuration and route files Per-request or per-instance pricing; quota, cost and tool proxying stay yours
A model-proxy AI gateway (Cloudflare, LiteLLM, Portkey, Kong, Azure) Model traffic, per each vendor's own documentation In the vendor's console or the proxy's config file A subscription or per-token fee; the team counters and the audit rows usually stay yours
SmartGate - an MCP-native algorithm gateway Token quota per key and team, cost accounting per task, audit and trace per call, tool proxying, alongside routing, auth and rate limiting In one policy object read on every request The platform fee, plus a share only after measured savings pass a threshold

The line to test against your own bill is pay for the platform, share only when you save: nothing is charged while measured savings stay under the threshold, and the Pro share is capped. Free covers 2,000,000 tokens a month with a 120-request-per-minute limit per key and 7-day logs; Pro starts at 18 dollars a month with 20,000,000 tokens and 30-day logs; Teams starts at 55 dollars with a 100,000,000-token pool and 90-day logs; Enterprise is contract pricing with a 180-day window. Current numbers: pricing page.

How to get started

  1. Write the responsibility table for your own stack. Nine rows, one owner each: routing, auth, request rate limiting, token quota, cost accounting, audit and trace, artifact paths, tool proxying, error shape.
  2. Keep your API gateway if it works. Routing, auth and rate limiting on a mature product are not the problem; put the missing rows behind it rather than in front of it.
  3. Put quota where the identity is resolved. A counter keyed by a team can be quoted in a review; a global counter cannot, and it cannot be attributed after the fact.
  4. Count tokens where the payload is parsed. A byte-level proxy can forward a call and cannot meter it.
  5. Record the refusal, not just the request. A capped request that leaves no row turns the next budget conversation into an archaeology project.
  6. Pick the split, then buy. Start on the free tier with start free - 2,000,000 tokens a month and all seven algorithmic primitives (smart_fetch, smart_search, smart_context_gate, smart_dedup, smart_budget_guard, smart_memory, smart_pipe). The tool surface is in the product docs, the quota model in token control, and the record in audit and compliance.

Frequently Asked Questions

Limitations and what this does not do

  • The shared layer is genuinely shared. Routing, authentication and request rate limiting are no argument for one product over another; the useful part of this page is the rows that are not shared.
  • The quoted code is one implementation. The excerpts come from a live product's publishing and audit paths. They show how these responsibilities are shaped in one codebase; they are not a specification, and no other product is claimed to match them.
  • An unknown status reads as an error. The status normaliser is conservative by design, so a new legitimate outcome counts as a failure until the whitelist is updated. Render the normalised value, but keep the raw one.
  • A missing count reads as zero. A task row whose token total was never written becomes zero in the summary, and a zero that means "not measured" will be misread as a saving.
  • The daily window is a UTC day. Fair as a fair-use ceiling, wrong for an invoice, and keyed per team rather than per key.
  • Two path directions, one review. A prefix map that falls back to the unknown kind and a parser that refuses unknown prefixes are deliberate opposites; both need the same owner, or the difference becomes a bug.
  • These are whole small functions, not whole files. Every excerpt is a complete short symbol, so the branches beside it - caches, response streaming, background jobs - are described rather than quoted.

Sources

Method note

The code here is not transcribed. Each block was cut out of the slice body returned by the SmartGate slice API and re-asserted byte-for-byte as a substring of that body; the first line inside every fence records the file and the source lines. Symbols are unique whole-name matches (rule A level 2) confirmed by slot-proof before reaching the prose, and every one of the eight planned sections matched a single symbol, so no section here is written from a guess. Every slice was short enough to quote whole, so no window skips a line in this round: the excerpts are complete functions and a complete class.

Demand figures come from this project's own DataForSEO Google Ads volume call, recorded in projects/ai-gateway-vs-api-gateway/search_volume.json and generated on 2026-09-21: the comparison phrase measures 20 searches a month at LOW competition (index 7, 70 in August), and the six mechanism phrases behind the code sections measure 170 each. No difficulty score is quoted, because the difficulty endpoint returned no row for this phrase. SERP details - 134 results, an AI Overview present with empty text and references, four People Also Ask questions, eight related searches, and the top organic domains - come from the round's own SERP record; nothing is claimed about the overview's contents, because there were none to read.

Slice provenance

# SERP keyword Symbol File Source lines How it was pinned sha256(12)
1 ai gateway vs api gateway hydratePseoBlocks lib/pseo/hydrate.ts 6–27 rule A L2 → slot-proof 40fd7b54366c
2 api gateway llm urlPrefix lib/pseo/kind-routing.ts 36–38 rule A L2 → slot-proof 869d582f429d
3 agent workflow memory publishPathFromKindSlug lib/pseo/kind-routing.ts 45–47 rule A L2 → slot-proof 48d4a3417f0d
4 anthropic model context protocol kindFromPublishPath lib/pseo/kind-routing.ts 66–81 rule A L2 → slot-proof 311529c46f17
5 mcp authorization normalizeTaskStatus lib/smartgate/audit-logs.ts 180–185 rule A L2 → slot-proof 35b24c6ab030
6 model context protocol docs transformAuditTask lib/smartgate/audit-logs.ts 187–198 rule A L2 → slot-proof 20e306785feb
7 rag vs agentic ai PlaygroundDailyCapError lib/smartgate/playground-daily-cap.ts 5–14 rule A L2 → slot-proof cbdb2cf4127b
8 token optimization dailyCapKey lib/smartgate/playground-daily-cap.ts 16–19 rule A L2 → slot-proof b41854b7ad09

Every fenced block above was cut from the slice body and re-asserted against it byte-for-byte before publication. 8 of 8 sections pinned, 0 abstentions, 0 misses. The comparison on this page is a map of responsibilities, not a claim that one layer replaces the other: routing, authentication and request rate limiting are shared ground, and the evidence above is about the rows that are not.