SmartGateSmartGate
Traffic Customs

Govern what agents fetch before it governs your bill

Uncontrolled fetch and search are the upstream cause of token bloat. Aggregate, extract, deduplicate, and compress — as one pipe or step-by-step.

The pain

Agents treat the web as free context. They re-fetch the same URLs, pile overlapping snippets into prompt windows, and pass the cost to your host LLM. Traffic customs means shaping tool output before it becomes model input.

Live pipe: research template

search→fetch→dedup→context_gate

From PIPELINE_TEMPLATES · backend/smartgate/core/pipeline.py

smart_search

Multi-engine snippets when no definitive URL exists. Don't use when user already gave a URL.

smart_fetch

HTTP(S) → Markdown-friendly content. Never invent page text.

smart_pipe

Orchestrate research · read · remember — or custom step arrays.

Pipe templates

TemplateStepsUse when
researchsearch → fetch → dedup → context_gateOpen-ended research
readfetch → context_gateUser supplied a URL
remembersearch → memory addPersist team facts across sessions

Before / after

ScenarioBeforeWith SmartGate
Research workflow4 ad-hoc tool callsOne smart_pipe research template
Page contentRaw HTML in contextMarkdown-friendly fetch output
Search overlapDuplicate passagesdedup step in research pipe

Traffic Customs as Research infrastructure

Uncontrolled fetch and search are the upstream cause of token bloat. Traffic Customs is how SmartGate governs what agents pull from the world before that material becomes model input. In Positioning Bible terms this is the Research capability—smart_search plus smart_fetch—composed with Context and Pipeline so a single intelligence runtime serves both human Web flows and agent MCP flows.

smart_search returns multi-engine snippets when the user has not already given a definitive URL. smart_fetch turns HTTP(S) pages into Markdown-friendly content without inventing text that was never on the page. smart_pipe orchestrates templates such as research (search → fetch → dedup → context_gate), read (fetch → context_gate), and remember (search → memory add), or custom step arrays your team defines.

Customs language is intentional: agents treat the open web as free context. They re-fetch the same URLs, pile overlapping snippets into prompt windows, and pass the cost to the host LLM. Shaping tool output at the gateway—aggregate, extract, deduplicate, compress—keeps the host model focused on reasoning instead of cleaning HTML. That is information intelligence governance, not another search API and not a model gateway.

Rate limits are part of the contract. smart_search and smart_fetch share a per-team gateway request budget separate from MCP requests-per-minute per API key. Free and Playground stay on strict fair-use entitlements; bursts soft-fail with HTTP 429 and Retry-After when available. Upgrade paths raise shared-workspace limits without changing the MCP endpoint your agents already call.

Adopt Traffic Customs when builders would otherwise rebuild fetch, extract, and orchestration inside every agent. One Streamable HTTP origin exposes the same primitives the dashboard meters. Human interfaces can run Smart Research scenes without showing MCP JSON; agent builders get the protocol surface on the agent-builders path and in Docs. Either way, the runtime underneath is the same intelligence layer between AI and the world.

When to search, fetch, or pipe

Prefer smart_fetch when the user already supplied a URL—searching first only adds redundant snippets. Prefer smart_search for open-ended questions with no canonical link. Prefer smart_pipe research when you want one audited chain instead of four ad-hoc tool calls. Prefer the read template for a single page cleanup; prefer remember when a fact should persist in team memory for later sessions.

Traffic Customs does not replace Perplexity-style research apps or LiteLLM-style model routers. SmartGate is research infrastructure and agent intelligence governance: the layer that decides how tool traffic is shaped, capped, and logged before your host model sees it. Pair with Token Control so long fetches are compressed and overlapping search hits are deduplicated, and with Audit & Compliance so every customs decision is traceable by team and request.

MCP example

{ "tool": "smart_pipe", "arguments": {
  "template": "research",
  "params": { "query": "MCP gateway enterprise adoption 2026" }
}}

Connect MCP in 5 minutes

Read MCP setup docs

Related roles and scenes

Same gateway, different jobs — pick the path that matches how you work.

Is / is not

Is: governed fetch and search (smart_fetch, smart_search) plus orchestrated pipes (smart_pipe) so agents return Markdown-friendly, deduped context instead of raw HTML piles.

Is not: a general web crawler SaaS, a browser automation product, or a substitute for your host LLM — SmartGate shapes tool traffic; your model still reasons.

See also Docs, Integrations, and About.

Tool rate limits

smart_search and smart_fetch each get 20 requests/minute per team — the cap is bucketed per tool, so one tool's traffic does not spend the other's — separate from MCP requests/minute per API key. See Docs for details.

Team memory

Team-scoped memory (add/search/get/delete). Session and team facts — not general world knowledge.

Related industries

Deeper guides on the same themes — from our industry library.

Browse all industries.