Govern what agents fetch before it governs your bill
Uncontrolled fetch and search are the upstream cause of token bloat. Aggregate, extract, deduplicate, and compress — as one pipe or step-by-step.
The pain
Live pipe: research template
From PIPELINE_TEMPLATES · backend/smartgate/core/pipeline.py
smart_search
Multi-engine snippets when no definitive URL exists. Don't use when user already gave a URL.
smart_fetch
HTTP(S) → Markdown-friendly content. Never invent page text.
smart_pipe
Orchestrate research · read · remember — or custom step arrays.
Pipe templates
| Template | Steps | Use when |
|---|---|---|
research | search → fetch → dedup → context_gate | Open-ended research |
read | fetch → context_gate | User supplied a URL |
remember | search → memory add | Persist team facts across sessions |
Before / after
| Scenario | Before | With SmartGate |
|---|---|---|
| Research workflow | 4 ad-hoc tool calls | One smart_pipe research template |
| Page content | Raw HTML in context | Markdown-friendly fetch output |
| Search overlap | Duplicate passages | dedup step in research pipe |
Traffic Customs as Research infrastructure
Uncontrolled fetch and search are the upstream cause of token bloat. Traffic Customs is how SmartGate governs what agents pull from the world before that material becomes model input. In Positioning Bible terms this is the Research capability—smart_search plus smart_fetch—composed with Context and Pipeline so a single intelligence runtime serves both human Web flows and agent MCP flows.
smart_search returns multi-engine snippets when the user has not already given a definitive URL. smart_fetch turns HTTP(S) pages into Markdown-friendly content without inventing text that was never on the page. smart_pipe orchestrates templates such as research (search → fetch → dedup → context_gate), read (fetch → context_gate), and remember (search → memory add), or custom step arrays your team defines.
Customs language is intentional: agents treat the open web as free context. They re-fetch the same URLs, pile overlapping snippets into prompt windows, and pass the cost to the host LLM. Shaping tool output at the gateway—aggregate, extract, deduplicate, compress—keeps the host model focused on reasoning instead of cleaning HTML. That is information intelligence governance, not another search API and not a model gateway.
Rate limits are part of the contract. smart_search and smart_fetch share a per-team gateway request budget separate from MCP requests-per-minute per API key. Free and Playground stay on strict fair-use entitlements; bursts soft-fail with HTTP 429 and Retry-After when available. Upgrade paths raise shared-workspace limits without changing the MCP endpoint your agents already call.
Adopt Traffic Customs when builders would otherwise rebuild fetch, extract, and orchestration inside every agent. One Streamable HTTP origin exposes the same primitives the dashboard meters. Human interfaces can run Smart Research scenes without showing MCP JSON; agent builders get the protocol surface on the agent-builders path and in Docs. Either way, the runtime underneath is the same intelligence layer between AI and the world.
When to search, fetch, or pipe
Prefer smart_fetch when the user already supplied a URL—searching first only adds redundant snippets. Prefer smart_search for open-ended questions with no canonical link. Prefer smart_pipe research when you want one audited chain instead of four ad-hoc tool calls. Prefer the read template for a single page cleanup; prefer remember when a fact should persist in team memory for later sessions.
Traffic Customs does not replace Perplexity-style research apps or LiteLLM-style model routers. SmartGate is research infrastructure and agent intelligence governance: the layer that decides how tool traffic is shaped, capped, and logged before your host model sees it. Pair with Token Control so long fetches are compressed and overlapping search hits are deduplicated, and with Audit & Compliance so every customs decision is traceable by team and request.
MCP example
{ "tool": "smart_pipe", "arguments": {
"template": "research",
"params": { "query": "MCP gateway enterprise adoption 2026" }
}}Connect MCP in 5 minutes
Read MCP setup docsRelated roles and scenes
Same gateway, different jobs — pick the path that matches how you work.
- Research and decisionsPersonal scene: search, fetch, dedup, and compress before the model answers.
- MCP for research and decisionsPersona guide: one MCP server entry, then governed research on every question.
- FinOps leadOwn the AI bill: per-team quota, alerts, and audit rows you can sign off on.
- Agent buildersWire one Streamable HTTP MCP endpoint — seven smart_* primitives, one runtime.
Is / is not
Is: governed fetch and search (smart_fetch, smart_search) plus orchestrated pipes (smart_pipe) so agents return Markdown-friendly, deduped context instead of raw HTML piles.
Is not: a general web crawler SaaS, a browser automation product, or a substitute for your host LLM — SmartGate shapes tool traffic; your model still reasons.
See also Docs, Integrations, and About.
Tool rate limits
smart_search and smart_fetch each get 20 requests/minute per team — the cap is bucketed per tool, so one tool's traffic does not spend the other's — separate from MCP requests/minute per API key. See Docs for details.
Team memory
Team-scoped memory (add/search/get/delete). Session and team facts — not general world knowledge.
Related industries
Deeper guides on the same themes — from our industry library.
- URL to Markdown
- Firecrawl alternative
- Agentic search vs RAG
- Agent pipeline
- Agentic workflow
- Agentic workflow examples
- AI workflow automation
- AI workflow automation tool
- AI agent orchestration
- MCP proxy
Browse all industries.