SmartGateSmartGate

MCP Server List: A Buyer's Checklist Before You Install

A list of MCP servers is only useful if you can evaluate one, and five questions do that: what the tool surface advertises, how a key is issued and checked, what limits the server can enforce, what the audit trail records, and where the counters live. A server that cannot answer all five is a prototype with an endpoint.

Short answer: A list of MCP servers is only useful if you can evaluate one, and five questions do that: what the tool surface advertises, how a key is issued and checked, what limits the server can enforce, what the audit trail records, and where the counters live. A server that cannot answer all five is a prototype with an endpoint.

Key takeaways

  • Read tools/list before you install anything. It is a contract you can inspect in one call, and it is the cheapest due diligence available.
  • Keys, not passwords. A server worthy of production issues credentials with a prefix, an expiry and a revocation path — and never stores the raw value.
  • Limits are only real if the counter is shared. A per-key window that lives in one process disappears with that process; the storage mode is part of the limit's meaning.
  • Audit is a feature you can test in five minutes. Trigger one call and look for the row.
  • A readiness endpoint that always says "ok" is not a readiness check. Ask what it actually verifies.
  • Do this next: run the checklist against one candidate server, and drop any that fail on the auth model regardless of how good the tool list looks.

Tool surface: what a server advertises before you install it

The first item on the checklist is the cheapest: call tools/list and read what comes back. A tool declaration is a contract — name, description, typed inputs, and hints about side effects:

# backend/smartgate/api/mcp.py — source lines 109–126 (smart_fetch)
@server.tool(
        name="smart_fetch",
        description=TOOL_DESCRIPTIONS["smart_fetch"],
        annotations=tool_annotations("smart_fetch"),
    )
    async def smart_fetch(
        url: str = Field(description="Full HTTP or HTTPS URL to fetch."),
        timeout: int = Field(default=30, description="HTTP timeout in seconds."),
    ) -> str:
        _, registry = _app_state()
        module = registry.get("fetch")
        ctx = _tool_ctx()
        return await _run_with_audit(
            "fetch",
            ctx,
            module.process(ctx, url=url, timeout=timeout),
            {"url": url},
        )

Read it the way an integrator would. The name is the vocabulary your agents will use, so vague names cost prompt tokens in every session. The description is written for a model, not for a marketing page. The inputs are typed with defaults, which is what lets a host fill them in without guessing. And the annotations say whether calling it can change state — the difference between an agent that asks before acting and one that does not. A server whose tool list you cannot fetch, or whose declarations carry no types, is one you will be debugging in your own prompts.

One thing the surface does not settle is direction: a tool list is what a caller reaches for, while an agent-to-agent hand-off carries a task rather than an invocation, and MCP vs A2A draws that line.

A typescript server tool: why the second declaration matters

One tool proves the server exists; the second proves it has a pattern. Vendors publish their list in every language, and the declarations should look like siblings:

# backend/smartgate/api/mcp.py — source lines 128–148 (smart_search)
@server.tool(
        name="smart_search",
        description=TOOL_DESCRIPTIONS["smart_search"],
        annotations=tool_annotations("smart_search"),
    )
    async def smart_search(
        query: str = Field(description="Search query string."),
        max_results: int = Field(
            default=10,
            description="Maximum number of results to return (1–50).",
        ),
    ) -> str:
        _, registry = _app_state()
        module = registry.get("search")
        ctx = _tool_ctx()
        return await _run_with_audit(
            "search",
            ctx,
            module.process(ctx, query=query, max_results=max_results),
            {"query": query},
        )

Same decorator, same description source, same dispatch shape, different arguments — a query and a result cap instead of a URL and a timeout. That regularity is the thing to check, because it predicts what happens at the tenth tool: a server that declares every tool through one code path will keep its schemas, descriptions and audit records consistent, while a server with ten hand-written handlers will drift. For a buyer, the practical test is to pick the two tools you will use most and confirm they return the same envelope. If they do not, everything downstream — retries, logging, cost attribution — becomes per-tool work.

Security starts with the key: how a credential is issued

The second checklist item is the auth model, and it starts at issuance rather than at verification. How a server mints a key tells you what it can later revoke:

# lib/api-keys/index.ts — source lines 43–69 (createApiKey)
async function createApiKey(
  teamId: string,
  userId: string,
  data: { name: string; expiresInDays?: number },
  planContext: { plan: Plan; features: Record<string, unknown> | null },
) {
  await assertCanCreateApiKey(teamId, planContext.plan, planContext.features);

  const { raw, hash, prefix } = generateKey();

  const expiresAt = data.expiresInDays
    ? new Date(Date.now() + data.expiresInDays * 86400000)
    : null;

  const key = await prisma.apiKey.create({
    data: {
      name: data.name,
      keyHash: hash,
      keyPrefix: prefix,
      expiresAt,
      teamId,
      createdById: userId,
    },
  });

  return { id: key.id, name: key.name, rawKey: raw, prefix };
}

Three properties are visible in one function. Issuance is plan-gated — the call checks the team's features before it writes anything, so a plan change can withdraw a capability rather than leaving it implicitly granted. The raw key is returned once and only the hash is stored, next to a display prefix, which means a database read does not hand out usable credentials. And expiry is optional and explicit: a key with no expiry never expires, which is a decision the buyer should make consciously rather than inherit. Ask any candidate server for those three properties; a vendor that keeps raw keys in a table has already failed the checklist.

What the key may then do on the wire is the other half of that model: OAuth and authorization covers the credential check, the bearer token and the team role behind it.

Authentication callbacks: what an identity object must contain

Behind the key is an identity, and the callback that produces it is the shape every later authorization decision reads:

# auth.ts — source lines 80–92 (authorize)
async authorize() {
              const testEmail = "demo@example.com";
              let user = await prisma.user.findUnique({ where: { email: testEmail } });
              if (!user) {
                user = await prisma.user.create({
                  data: { email: testEmail, name: "Demo User", emailVerified: new Date(), role: "USER" },
                });
                await prisma.account.create({
                  data: { userId: user.id, type: "credentials", provider: "demo", providerAccountId: user.id },
                });
              }
              return { id: user.id, email: user.email, name: user.name, image: user.image, role: user.role };
            }

This excerpt is the development credential provider from the project's own auth configuration: it looks up a demo address, creates the user and its account link if they are missing, and returns an identity object. Ignore the hard-coded address — it is a stub for local sign-in. What the stub shows is the shape that matters when you evaluate a vendor: an internal id, an email, a display name, an avatar and a role. Every limit, quota and audit row downstream is attributed to one of those fields, so a server that returns an anonymous session has nowhere to hang its per-user accounting. Ask what the identity object contains before you ask how the login page looks.

Hosting: where the counters live decides the limits

Item three — limits — has a hosting prerequisite that most checklists put on the wrong line. A limit is a counter, and a counter needs a store the server can reach from every process:

# lib/redis/config.ts — source lines 26–29 (isTcpRedisConfigured)
function isTcpRedisConfigured(): boolean {
  const url = redisUrl();
  return Boolean(url && (url.startsWith("redis://") || url.startsWith("rediss://")));
}

The check is narrow on purpose: it asks whether a TCP-speaking store is configured on either of the two connection schemes, and returns a boolean rather than throwing. That narrowness is what lets an operator tell three states apart — nothing configured, something configured in a shape this path cannot use, and ready. For a buyer, the question to ask is concrete: when my key is rate-limited, is that decision made against shared state or against one process's memory? The answer is the difference between a limit that holds across a fleet and one that multiplies by replica count.

The openclaw case: a degraded mode is still a mode

The follow-up question is what happens when that store is missing, because the honest answer is usually "the server keeps running":

# lib/redis/config.ts — source lines 38–42 (resolveRedisMode)
function resolveRedisMode(): RedisMode {
  if (isUpstashRestConfigured()) return "upstash";
  if (isTcpRedisConfigured()) return "tcp";
  return "none";
}

Two checks in priority order and a constant. A REST-based store wins, then a TCP connection, then none. Those are not three equivalent configurations: the first two enforce limits across processes, and the third admits everything while the audit layer still records it. Read that as a specification of the failure mode rather than a defect. An evaluator should insist on knowing which of the three a given deployment runs in, because a hosted server that silently resolved to none has per-key limits that exist only in its documentation — and the way to find out afterwards is that the audit rows keep appearing while the spend does not stop.

The tutorial step: what a client must paste

Item four is the install itself, and the right question is not "is there a quickstart" but "how much of it is generated". A server that renders its own client configuration has one source of truth:

# lib/connect/mcp-config-templates.ts — source lines 161–181 (buildGenericMcpConfigJson)
function buildGenericMcpConfigJson(
  mcpUrl: string,
  apiKeyPlaceholder: string = API_KEY_PLACEHOLDER,
): string {
  return JSON.stringify(
    {
      mcpServers: {
        smartgate: {
          type: "streamable-http",
          url: mcpUrl,
          headers: buildMcpAuthHeaders(
            apiKeyPlaceholder,
            PLATFORM_AGENT_ID.Generic,
          ),
        },
      },
    },
    null,
    2,
  );
}

Four fields do the work: the transport type, the endpoint URL, the auth header and the server name. Because the block is generated rather than written by hand, the placeholder in the header is the same placeholder the server validates, and the field names match what the endpoint accepts. For a buyer, the testable version of that claim is simple: paste the block the vendor generates, restart the host, and confirm that the first call arrives — not that the tutorial's screenshot matches your version of the client. A tutorial that requires hand-editing is a maintenance cost you will pay on every host you add. The endpoint contract itself is on the MCP endpoint page.

If the host you are pasting into is Claude Desktop, the client half of that same contract — which hosts Anthropic ships and what sits between them and a hosted server — is on Anthropic MCP.

Audit and events: the example of a signed delivery

Item five is the audit trail, and it has two halves: what the server records, and what it pushes out. A server that delivers events should sign them per endpoint:

# lib/webhooks/index.ts — source lines 41–63 (createEndpoint)
async function createEndpoint(
  teamId: string,
  data: { url: string; description?: string; events?: string[] },
  planContext: { plan: Plan; features: Record<string, unknown> | null },
) {
  await assertCanCreateWebhookEndpoint(
    teamId,
    planContext.plan,
    planContext.features,
  );

  const secret = `whsec_${randomBytes(24).toString("hex")}`;

  return prisma.webhookEndpoint.create({
    data: {
      url: data.url,
      secret,
      description: data.description,
      events: data.events ?? [],
      teamId,
    },
  });
}

The excerpt is the outbound half. An endpoint is created only if the plan allows it, and it is issued its own secret — 24 random bytes behind a recognisable prefix — so a delivery can be verified by the receiver rather than trusted. Both properties are worth asking about, because webhook delivery is one of the two ways an audit trail leaves the vendor's console. The other way is the per-call record the server writes when a tool runs; trigger one call against a candidate server and look for the row. If the call succeeded and no row exists, the audit claim is decoration. The read side of that picture is documented on the logs page.

What the row itself owes you — the fields it keeps, the retention window that ends it and the questions it cannot answer — is worked through on MCP logging and observability.

Remote server health: the readiness check to ask for

The last item is the one buyers skip: what the server says about itself when it is degraded. A health endpoint is only useful if it can fail:

# backend/smartgate/main.py — source lines 225–232 (health)
@app.get("/health")
async def health():
    ready = audit_logger.is_ready
    return {
        "status": "ok" if ready else "degraded",
        "audit_logger_ready": ready,
        "modules": list(registry.modules.keys()),
    }

Eight lines and one honest field: the endpoint reports ok only when the audit logger is ready, and degraded otherwise, and it lists the modules the process actually loaded next to that verdict. That is what a monitoring system needs to page on something real rather than on a 200 that always says fine. When you evaluate a hosted server, ask what its readiness endpoint checks and what it returns when a dependency is missing. A service whose health check only proves the process is listening will report healthy through exactly the incident you care about. For a remote deployment, this is the item that decides whether your own on-call rotation can trust a green dashboard.

A python mcp server: what you would build yourself

Some teams should not buy an MCP server at all, and the checklist should say so. A Python server in the official SDK is a few dozen lines: create the server object, declare tools with decorators, run it over stdio for a single local client or mount a Streamable HTTP app for several. Nothing in this protocol requires a hosted product, and for one process serving one client the local option is strictly cheaper. The list changes shape the moment any of the following is true: more than one host, more than one team, credentials that need revoking, limits that need enforcing, or a spend report anyone wants to read. Those are the four questions that turn a weekend script into a hosting decision — and the checklist above is what you apply to whichever server you end up installing.

The mcp server tutorial is that build walked through end to end, which is the quickest way to see which of the four questions you would still be answering six months in.

Building from scratch: what the checklist does not cover

Building your own server does not remove the checklist; it moves every item into your backlog. You will still need an auth model, because stdio servers on a shared machine have none worth the name. You will still need a counter store if more than one process answers calls, and a readiness endpoint if anyone will page on your service. You will still need an audit record, because "which agent made this call" is the first question in every incident review. What a hosted server sells is not the protocol implementation — that part is genuinely small — but the answers to those five questions already written, tested and observable. Read the MCP server page for the implementation half; the checklist is what you keep either way.

The implementation half is genuinely small — a model context protocol server is a few hundred lines — which is exactly why the checklist, not the code, is the part that decides whether the install survives its second month.

Testing before you install: the acceptance run

Every item above can be checked in one sitting, and the order matters. Start by fetching tools/list and confirming that the declarations are typed and annotated. Then create a key, confirm it is shown once, and verify that a second list call with a revoked key fails. Trigger one tool call and find its audit row. Ask which storage mode the deployment resolves to, and what the readiness endpoint returns with the store unreachable. Then paste the generated config into one host and watch the first request arrive. That sequence takes under an hour and it exercises the whole product: the tool surface, the auth model, the limits, the audit trail and the hosting assumptions. A server that survives it is one you can install; a server that fails any step is one that would have failed in production instead.


How SmartGate compares

Checklist item What to ask the vendor SmartGate's answer Where to verify it
Tool surface Fetch tools/list: are the declarations typed and annotated? Seven smart_* tools, each with a title and a read-only hint MCP tools reference
Auth model Is the raw key stored? Can it expire and be revoked? Keys are shown once, stored as a hash with a prefix, and carry an optional expiry security page
Limits Per key, per team, or per process? Per-key window plus a team ceiling, clamped to the plan MCP gateway
Audit Does every tool call leave a row? One audited exit for all seven tools, plus signed webhook delivery logs page
Hosting What happens when the counter store is unreachable? The storage mode is resolved at boot and visible; limits degrade rather than block connect page

The pattern in that table is worth naming: every row is a question with a checkable answer. That is the difference between a list of MCP servers and a shortlist. Pricing belongs in the same exercise — the pricing page lists the free tier (2M tokens a month, 120 MCP requests a minute per key) next to the paid limits, so cost per call is a number you can compute rather than a sales conversation.

How to get started

  1. Run the checklist on one candidate. Tools, keys, limits, audit, hosting — in that order.
  2. Attach it to one host using the connect page; if the vendor generates the block, use theirs rather than editing by hand.
  3. Call the server once with a harmless tool and confirm the result envelope matches what the declarations promised.
  4. Find the audit row. If it is missing, stop — the rest of the checklist does not matter.
  5. Compare before you commit: the endpoint model in MCP vs REST API is the alternative most teams are actually choosing between.

Start on the free tier — 2M tokens a month, all seven tools, 120 MCP requests a minute per key — and read the pricing page before you budget a paid tier.

Frequently Asked Questions

How many MCP servers should I evaluate?

Three is usually enough: one hosted product, one self-hosted option and the server you would write yourself. The checklist takes an hour per candidate, and it discriminates quickly.

What is the first thing to check on an MCP server?

The tool surface. Fetch tools/list and read the declarations — if the names are vague, the inputs untyped or the side-effect hints missing, nothing further down the checklist will be pleasant.

Can an MCP server work without an API key?

Yes, and for one local process and one client that is normal: a stdio server inherits the trust of the machine that launched it. The moment more than one caller reaches it, the auth model matters.

Do rate limits work if the server has no shared store?

Not per key across processes. Without a store the limit is enforced per process, so a second instance doubles the allowance; a deployment with no store at all admits everything.

How do I verify the audit trail?

Trigger one tool call and look for the row, then revoke the key and confirm the call fails. A vendor that cannot show you a row for a call you just made does not have an audit trail.

Is a hosted server faster than a local one?

No, and speed is the wrong axis. A local server avoids the network hop; a hosted one gives you credential revocation, shared limits and an audit record your team can read.

Limitations and what this does not do

  • This is an evaluation checklist, not a ranking. No vendor is endorsed here, and the order of the items reflects what breaks first in production rather than what is most marketable.
  • The excerpts are one implementation. Every claim about how keys, limits, audit rows or config blocks should work is illustrated with SmartGate's own code; another server may satisfy the same checklist differently.
  • One excerpt is a development stub. The auth callback quoted above is the local sign-in provider, kept for the shape of the identity object it returns rather than as a credential check.
  • Nothing here tests performance or accuracy. Latency, model quality and retrieval quality are separate questions, and this checklist will not tell you whether a tool's answers are any good.
  • Compliance frameworks are out of scope. Retention windows and audit completeness are properties a deployment can evidence; no checklist substitutes for your own review.
  • These are windows, not whole files. Each excerpt is a slice of a longer module, so surrounding validation and error handling are described rather than quoted.

Sources