At nine on a Monday the roof-data provider behind the assessment service starts answering slowly. Nothing is down. Calls that took a fraction of a second now take many seconds, and some never return.
Within minutes all four teams notice. The claims assistant's calls hang, so its host retries them, so the assessment service receives three calls for every one it received before. The support desk's status lookups, which do not even need the provider, queue behind them for the same worker threads. Analytics jobs time out. And the underwriting agent's call to record_recommendation for SUB-1004 times out on the agent's side; the agent retries, and an earlier design of the service would have stored two recommendations for one submission, one of which no underwriter would ever see linked to a decision.
Every component behaved as it was built to. The provider was slow, the clients retried, the server accepted work. What made it an outage was that four teams shared one server and nothing limited how much of it one failure could take. The series case UW-006 (provider timeout) and this site's own case UW-M08 are both versions of this morning.
Why a shared server fails all at once#
A function inside one application fails alone: if the roof lookup hangs, one application hangs. An MCP server shared by four teams pools their fate. Its worker threads, its connections to the assessment service and the service's own capacity are shared, so the slowest dependency sets the pace for everyone, and the most aggressive client takes the most of what is left.
Three forces turn slowness into collapse. Waiting holds resources: a call that waits thirty seconds holds a worker for thirty seconds. Retries multiply load: a host that retries three times and a server that retries three times inside each attempt send nine calls for one. And writes repeated by a retry change data: a timeout says nothing about whether the write happened.
Before AI. A backend engineer has fought this behind an API gateway. A Spring service calling a slow downstream over a client with no timeout exhausted its thread pool, and then its own callers' pools. The fixes were the same ones on this page: a timeout on every outbound call, retries only where safe, an idempotency key on payment writes, a rate limit per API key at the gateway, a breaker that stops calling a dependency that is down. MCP changes none of these; it adds a new kind of caller, an agent, that retries tirelessly unless something says stop.
The smallest primitive: a deadline#
A timeout is the longest time one attempt may take before the caller gives up on it. For example, the example gives one call to the assessment service 1000 ms (simulated). A deadline is the same idea for the whole call, all attempts included: the example gives a tools/call 3000 ms from the moment the server starts it. Without a deadline, retries can extend a call forever; with one, every retry must fit inside the remaining time.
The example keeps these numbers in one record, RetryPolicy in examples/uw-mcp/resilience.py, verbatim:
@dataclass(frozen=True)
class RetryPolicy:
max_attempts: int = 3 # for reads; writes always 1
per_attempt_timeout_ms: int = 1000
deadline_ms: int = 3000
backoff_base_ms: int = 200
attempt_cost_ms: int = 40 # simulated time a successful attempt takes
Every time in the example runs on a simulated clock, so these are configuration values in a teaching program, not measurements and not advice for real latencies.
Normative (MCP specification 2026-07-28): implementations should establish timeouts for all sent requests, to prevent hung connections and resource exhaustion; when a request has no response within the timeout, the sender should cancel it and stop waiting. On Streamable HTTP, closing the response stream is the cancellation, and the server must treat a client disconnect as cancellation of that request. On stdio, the client must send notifications/cancelled with the request id. Implementations may reset the timeout on a progress notification but should always enforce a maximum. And because a cancellation can arrive after the work finished, both sides must handle that race. The tools page adds that clients should set timeouts on tool calls.
The race matters most for writes. A cancelled record_recommendation may already have committed. Cancellation stops waiting; it does not undo.
Bounded retries, for reads only#
A retry is worth trying when the failure is likely to be brief and repeating the call cannot change data. A bounded retry is a retry with three limits: a maximum number of attempts, a growing wait between them (backoff) and the deadline over all of them. For example, the example retries a read at most three times, waits 200 ms after the first failure and 400 ms after the second, and never starts an attempt that cannot fit before 3000 ms.
The loop is call_with_retry in resilience.py. This excerpt is the part that decides whether to retry, verbatim except that the function body's indentation is removed:
max_attempts = policy.max_attempts if read_only else 1
timeline: list[dict[str, Any]] = []
last = "timeout"
for attempt in range(1, max_attempts + 1):
remaining = policy.deadline_ms - (clock.now_ms - start)
if remaining <= 0:
break
The first line carries the whole policy: the server retries reads, never writes. Case UW-M08 makes the assessment service time out on every attempt of get_roof_assessment("SUB-1004"). Run in a scratch copy on 2026-11-18, the server's spans show what the policy did:
--- UW-M08 read {"http_status": 200, "ok": false, "code": "timeout", "error": {"code": "timeout", "message": "The assessment service did not answer within the deadline (3 attempts).", "retryable": true, "attempts": 3}}
--- UW-M08 read spans
{"span_id": "c834a9a6f36facce", "parent_span_id": "09ebf646f776e265", "name": "tools/call get_roof_assessment", "start_ms": 0, "end_ms": 3000, "status": "error"}
{"span_id": "46b8be9a0e40311f", "parent_span_id": "c834a9a6f36facce", "name": "authorize", "start_ms": 0, "end_ms": 0, "status": "ok"}
{"span_id": "7c89e2bb869d7606", "parent_span_id": "c834a9a6f36facce", "name": "service.get_roof_assessment", "start_ms": 0, "end_ms": 3000, "status": "error"}
{"span_id": "df5032bd2003cba1", "parent_span_id": "7c89e2bb869d7606", "name": "attempt 1", "start_ms": 0, "end_ms": 1000, "status": "timeout"}
{"span_id": "b4e4e3e6342007d1", "parent_span_id": "7c89e2bb869d7606", "name": "attempt 2", "start_ms": 1200, "end_ms": 2200, "status": "timeout"}
{"span_id": "0c077483899833c4", "parent_span_id": "7c89e2bb869d7606", "name": "attempt 3", "start_ms": 2600, "end_ms": 3000, "status": "timeout"}
Attempt 1 used its full 1000 ms. The server waited 200 ms, tried again, waited 400 ms, and started attempt 3 at 2600 ms with only 400 ms left, so attempt 3 was cut at the deadline. At 3000 ms the server stopped and returned a controlled error. The caller waited three seconds (simulated), not three attempts times a full timeout plus waits, and the service saw three calls, not an open-ended stream. The model here is a scripted stand-in, so these results show how the protocol and the controls work, not how good any model is.
The example's backoff has no random spread (jitter), so two runs give the same bytes. A production retry adds jitter, so a thousand clients that failed together do not retry together. How a single tool's retries, timeouts and breakers are tuned in depth is taught on tools.sammuthu.com: reliability; this page stays with what changes when the server is shared.
Why a write is attempted once, and what makes the retry safe#
Repeat a read and you get the same facts. Repeat record_recommendation and, without protection, you get a second recommendation. Worse, a timeout does not tell the caller which case it is in: the write may have failed before the database, or committed while the reply was lost on the way back (Client and server calls this partial failure). So the server does not retry writes at all (max_attempts is 1 for a write), and the safety of a retry by the client comes from the data, not from the loop.
The agent's application generates one key per intended write and sends it with every attempt of that write. The assessment service stores the key with the row, in a column with a UNIQUE constraint. When the same key arrives again with the same payload, the service returns the stored row instead of inserting another. That key is called an idempotency key; in the example it is SUB-1004:run-7f3a:recommendation: the submission, the agent's run and the action.
Case UW-M08 continues after the read. The service times out once on the write; the client retries with the same key; then the client retries again, as if the reply to the second attempt had been lost.
Diagram: UW-M08 · a write times out, the client retries with the same key, one row
Time runs down the page. The host's MCP client sends record_recommendation with an idempotency key. The server makes exactly one attempt; the service times out, and the server returns a controlled timeout error marked retryable. The client sends the same call again with the same key: the service inserts the row. The reply to that call is lost, so the client sends it once more: the service finds the key, compares the payload hash and returns the stored row with idempotent_replay true. Three requests, one row.
- Host's MCP client to MCP server uw-assessment: tools/call record_recommendation · key. JSON-RPC id 2. The key names the submission, the run and the action.
- MCP server uw-assessment to Assessment service: attempt 1 (only attempt). The service does not answer within the deadline; nothing was inserted in this run.
- MCP server uw-assessment to Host's MCP client: isError · timeout · retryable. A controlled execution error with the number of attempts.
- Host's MCP client to MCP server uw-assessment: same call, same key. JSON-RPC id 3. A new request, the same intended write.
- Assessment service to recommendations table: insert, key UNIQUE. The row is stored with the key and the payload hash in one transaction.
- MCP server uw-assessment to Host's MCP client: reply lost on the way back. The write committed, but the client cannot know: the case a timeout always leaves open.
- Host's MCP client to MCP server uw-assessment: same call, same key, again. JSON-RPC id 4.
- Assessment service to Host's MCP client: stored row · idempotent_replay true. Same key, same payload hash: the stored row is returned and nothing is inserted.
The run output shows the same three requests:
--- UW-M08 write id=2 {"http_status": 200, "ok": false, "code": "timeout", "error": {"code": "timeout", "message": "The assessment service did not answer within the deadline (1 attempts).", "retryable": true, "attempts": 1}}
--- UW-M08 write id=3 {"http_status": 200, "ok": true, "code": null, "result": {"recommendation_id": "REC-0001", "submission_id": "SUB-1004", "recommendation": "escalate", "status": "awaiting_underwriter", "idempotent_replay": false}}
--- UW-M08 write id=4 {"http_status": 200, "ok": true, "code": null, "result": {"recommendation_id": "REC-0001", "submission_id": "SUB-1004", "recommendation": "escalate", "status": "awaiting_underwriter", "idempotent_replay": true}}
rows 1
One attempt for the write, then a client retry that stored REC-0001, then a replay of the same row with idempotent_replay true. One row. The key protects against the opposite mistake too: the same key with a different payload is refused, because it means the client is confused about what it is writing.
--- same idempotency key, different payload
{"http_status": 200, "ok": true, "code": null, "result": {"recommendation_id": "REC-0001", "submission_id": "SUB-1004", "recommendation": "escalate", "status": "awaiting_underwriter", "idempotent_replay": false}}
{"http_status": 200, "ok": false, "code": "conflict", "error": {"code": "conflict", "message": "This idempotency key was used with a different recommendation.", "retryable": false}}
rows 1
Common confusion. The MCP tool annotationidempotentHintdoes not make a tool idempotent. The specification (revision2026-07-28, normative) calls annotations hints, and clients must consider them untrusted unless they come from a trusted server. Idempotency is a property of the domain service and its data; the NSA's MCP guidance notes that MCP does not enforce it.
What the model is told#
When a protection fires, someone must decide what happens next, and the reply should help them decide well. Normative (MCP specification 2026-07-28): tool execution errors, including API failures, go in the result with isError true, and clients should pass them to the model so it can self-correct; unknown tools and malformed requests are protocol errors.
The example's results for these cases are execution errors with a small JSON body: a code, a message, retryable, and for timeouts the number of attempts. retryable is true only for timeout, rate_limited and unavailable. A refusal or a conflict is not retryable, and saying so stops a model from looping on a call that will never succeed.
This site's recommendation: the host application, not the model, owns the retry policy for writes. The model may decide to try a read again later or to tell the user the evidence is unavailable; it should never generate a new idempotency key for the same intended write, because that turns a safe retry into a second recommendation. If the model must retry a write, the host reuses the key it already holds.
One team's burst must not become everyone's outage#
Retries and deadlines limit the damage of one slow dependency. They do nothing about one fast client. If the claims assistant sends a burst of calls, it takes worker threads, service capacity and provider quota that support and analytics need.
A rate limit caps how many calls one consumer may make in a period. The example implements it per client id as a token bucket: each client has a bucket of tokens, each call takes one, and tokens refill at a fixed rate up to the bucket's size. For example, the claims assistant's bucket holds 3 and refills 1 per second, so three quick calls pass and a fourth within the same second waits. Run on the simulated clock:
--- token bucket: capacity 3, refill 1 per second (claims-assistant)
0 {"ok": true, "retry_after_ms": 0}
100 {"ok": true, "retry_after_ms": 0}
200 {"ok": true, "retry_after_ms": 0}
300 {"ok": false, "retry_after_ms": 700}
1300 {"ok": true, "retry_after_ms": 0}
The fourth call at 300 ms finds 0.3 of a token and is told to come back in 700 ms. Over the example's HTTP server, that call returns a tool execution error, and the audit store records why:
200 {"resultType": "complete", "content": [{"type": "text", "text": "{\"error\": {\"code\": \"rate_limited\", \"message\": \"Client claims-assistant is over its rate limit; try again later.\", \"retryable\": true, \"retry_after_ms\": 700}}"}], "isError": true, "_meta": {"io.modelcontextprotocol/serverInfo": {"name": "uw-assessment", "version": "0.1.0"}}}
{"event_id": "AUD-0004", "who": {"sub": "agent:claims-assistant", "principal_type": "agent", "client_id": "claims-assistant", "business_unit": "commercial-west", "jti": "jti-820ab67b"}, "decision": {"result": "refused", "reason": "rate_limited", "check": "rate_limit", "detail": "client claims-assistant is over its limit"}, "outcome": "rate_limited"}
Normative (MCP specification 2026-07-28): servers must rate-limit tool invocations. The specification read here does not say how a rate-limited call is answered. The example's choice is an execution error with retry_after_ms, so the model and the host see it; the other common choice is an HTTP 429 Too Many Requests from a gateway in front of the server (RFC 6585, which lets the answer carry a Retry-After header), and then the host's MCP client sees it before the model does.
The shared lab opens on the rate-limit panel and the call timeline. It replays the scripted workload of all four teams against the example's server and shows each team's bucket and metrics apart.
Synthetic counts from a scripted workload, in this example; they measure nothing.
| Team | Client | Scopes | Visible tools | Calls / allowed / refused / rate limited | Bucket |
|---|---|---|---|---|---|
| claims | claims-assistant | uw.read | get_roof_assessment, search_guidance, check_eligibility | 6 / 4 / 1 / 1 | 3, refill 1 per s |
| underwriting | uw-workbench | uw.read uw.recommend | the three reads and record_recommendation | 4 / 4 / 0 / 0 | 10, refill 5 per s |
| underwriting | uw-cockpit (the underwriter) | uw.read uw.decide | the three reads and record_underwriter_decision | 1 / 1 / 0 / 0 | 5, refill 1 per s |
| support | support-desk | support.read | get_assessment_status | 3 / 2 / 1 / 0 | 5, refill 2 per s |
| analytics | analytics-jobs | analytics.read | get_assessment_stats | 4 / 2 / 1 / 1 | 2, refill 1 per s |
The claims assistant's fourth call within 300 ms found its bucket empty, and analytics' third statistics call within 20 ms found its bucket of 2 empty. The underwriting workbench, with its own bucket, made four calls in 150 ms and was never slowed. That is the whole point of a limit per consumer: the claims burst spent the claims budget, not the underwriting team's.
One detail of the example deserves attention: the rate limit is checked first in McpServer.tools_call, before schema validation and authorization. A flood of calls that would be refused still spends the caller's tokens, so a misconfigured client cannot make the server do unlimited authorization work. The cost is that a refused call counts against the limit; for a shared server that is the right trade.
When the dependency is down, stop calling it#
A deadline caps how long each call waits for a dead service; it still waits. When the assessment service is down, every team's calls spend their full deadline failing, and the service, if it is trying to recover, receives a steady stream of calls.
A circuit breaker counts failures in a row and, past a threshold, stops sending calls for a cool-down period: calls fail at once with circuit_open. After the cool-down it lets one call through (half-open); success closes the circuit, failure opens it again. For example, the example's breaker opens after 5 failures and cools down for 30000 ms. Run with the service unavailable:
--- circuit breaker: threshold 5, cool-down 30000 ms; service unavailable
1 40 False unavailable closed
2 80 False unavailable closed
3 120 False unavailable closed
4 160 False unavailable closed
5 200 False unavailable open
6 200 False circuit_open open
after cool-down 30240 True 1 closed
transitions [{"t_ms": 200, "from": "closed", "to": "open"}, {"t_ms": 30200, "from": "open", "to": "half_open"}, {"t_ms": 30240, "from": "half_open", "to": "closed"}]
The sixth call took no time at all: it failed at 200 ms, the same moment it started. After the cool-down, the first call went through and closed the circuit.
The breaker in the example belongs to the server's gateway to the assessment service, so there is one breaker for that dependency, shared by every team. That is right for a dependency that is down for everyone. It is wrong when the failure is specific: if only commercial-east submissions fail because one database shard is down, a shared breaker would also stop commercial-west calls. The folded section below compares the choices.
The client's fallback, and what it must never do#
When the server answers timeout, circuit_open or rate_limited, the host still has a user and a task. A fallback is what the host does instead of the failed call. For example, the underwriting workbench marks SUB-1004 "roof evidence unavailable; retry later" and lets the agent's recommendation say that the roof age could not be confirmed.
A fallback must degrade honestly. The tempting one in this domain is to use the roof age the application stated (9 years) when the provider (22 years) is unreachable. That is a fabricated fact: it is exactly the gap R-ROOF-03 exists to catch, and the recommendation would look confident while hiding the failure. This site's recommendation: a fallback may return stale data clearly marked with its age, a reduced answer, or a request for information (the shape of R-ROOF-02, roof age unknown), and it must leave the decision path untouched. The model recommends; a person decides, and a person deciding on hidden missing evidence is not deciding.
The order of protections at a shared server#
Put together, each protection sits at a specific point on the path of a call, and the order matters. The example's McpServer.tools_call applies them in this order:
Diagram: The order of protections on one tools/call · cheapest first, the service last
Read left to right; this is the order in McpServer.tools_call in the example. The per-client rate limit runs first, before any other work. Schema validation and authorization follow, so no refused or malformed call reaches the service. At the hop to the service, the circuit breaker fails fast when the dependency is down, and the deadline with bounded retries wraps each attempt for reads. Writes get one attempt; their retry safety lives in the service, behind the idempotency key. Every exit writes an audit event and returns a result the model can read.
- Admit · per consumer
- Rate limit: One bucket per client id from the validated token; checked before anything else.
- Check · per call
- Schema: Malformed arguments end here as invalid_arguments.
- Authorize: Refused calls never reach the service.
- Call · per dependency
- Circuit breaker: Fails at once with circuit_open while the dependency is down.
- Deadline + bounded retry: Timeout per attempt, backoff 200 then 400 ms, no attempt past the deadline. Writes: one attempt.
- Service · owns the data
- Assessment service: A repeated key with the same payload returns the stored row; another payload is a conflict.
- Answer · host and model
- Result or isError: The host decides: retry a read later, reuse the key for a write, or fall back honestly.
The rate limit comes first because it is the cheapest and protects everything after it. Schema validation and authorization come next, so no refused or malformed call reaches the service. The breaker sits at the hop to the service, and the deadline and retries wrap each attempt. Writes skip the retry; their safety lives in the service, behind the idempotency key.
What goes wrong without these ideas#
| Missing idea | What happens |
|---|---|
| Per-call deadline | A slow provider holds every worker; support waits behind claims |
| Bounded retries, reads only | Host and server retries multiply; a write retried by the server can store two rows |
| Idempotency key on writes | The client's retry after a lost reply records a second recommendation |
| Rate limit per consumer | One team's burst spends the shared capacity of all four |
| Circuit breaker | Every call to a dead service waits out its full deadline |
| Honest fallback | The agent substitutes the application's roof age and hides the failure |
| Retryable flag in the error | The model loops on a refusal or a conflict |
Security and governance notes#
Security note. Rate limits are a security control as well as a capacity control: an agent tricked by injected text into calling a tool in a loop is stopped by its client's bucket, and the audit events show the pattern. Keep limits per client id from the token, never from clientInfo, which the client reports about itself. And never let a fallback widen access: a "degraded mode" that skips the authorization check when the policy service is slow is an outage turned into a breach.
Governance note. Limits and breaker thresholds are part of the service contract with each consuming team: the platform team sets and publishes them in the catalog, the consuming application owner agrees to them, and a change is announced like a schema change. Who approves a raised limit for one team, and who is paged when a breaker opens, belongs in the RACI on Governance.
In a coding agent. A coding agent that calls a shared repository server meets the same limits.search_codeis a read and can be retried with backoff;open_pull_requestis a write and needs a key such as the branch and the commit, so a retried call returns the pull request already opened instead of opening a second one. A local stdio server fails alone with the developer's laptop; a company-wide remote server fails for every developer at once, which is why it needs per-team rate limits. Harness Engineering covers the agent runtime that decides when to stop retrying.
What this is not. This page covers what a shared MCP server must do so one failure does not become everyone's. Per-tool timeouts, retries and breakers in depth belong to tools.sammuthu.com: reliability, and recovery of a whole agent system (checkpoints, resumption, compensation) belongs to https://reliable.sammuthu.com.
Checklist for Monday#
| Check | Pass when |
|---|---|
| Deadline | Every tools/call has a deadline over all attempts; every outbound call has a timeout |
| Retries | Reads retry with backoff, jitter and a small maximum; writes are never retried by the server |
| Keys | Every write tool takes an idempotency key; the service stores it UNIQUE with a payload hash |
| Conflict | The same key with another payload is refused, not overwritten |
| Rate limits | One bucket per client id from the token; the answer says when to come back |
| Breaker | Each dependency has a breaker; its state changes are recorded |
| Errors | Execution errors carry a code, a message and retryable; refusals are not retryable |
| Fallback | Degraded answers are marked; no fallback invents evidence or skips authorization |
Exercises#
- The claims host retries every failed call three times, and the server retries each read three times. How many calls can one failed claims lookup send to the assessment service, and what do you change?
- A developer proposes that the server retry
record_recommendationtoo, "since the idempotency key makes it safe". What is wrong with the argument? - Analytics asks for a bucket of 50 for a nightly job. Which owners decide, and what else should change at the same time?
AnswersOne line each
- Up to twelve (four host attempts, each with three server attempts) within the host's own timeout; retry in one layer only (the server, inside its deadline) and have the host respect
retryableand the deadline. - The key makes a client's retry safe, but a server retry inside one deadline can overlap an attempt that is still committing and doubles the load on a struggling service; one attempt per call keeps the outcome readable.
- The platform team sets limits with the consuming application owner, recorded in the catalog; give the job its own client id and schedule, so its burst cannot spend the interactive teams' capacity.
Retry storms across layers and teamsRetries multiply by layer and add by team; a budget, not a count, keeps them bounded
Count the layers that can retry a single roof lookup: the model may ask again, the host's agent loop may re-run the step, the host's MCP client may retry the request, a gateway may retry the upstream call, and the server retries the service call. Three attempts at each of three layers is twenty-seven calls for one lookup. Multiply by four teams that all saw the same slowdown at the same moment, and the assessment service receives a wave larger than its normal traffic at the moment it is least able to serve it.
Two designs bound this. The first is "retry in one layer": the server retries reads inside its deadline, and every other layer passes timeout through without retrying, except the host deciding later, at a human pace, to try the task again. The second is a retry budget: each client may spend at most a small fraction of its calls on retries over a window, and once the budget is spent, failures are returned at once. A budget adapts to scale; a fixed count does not.
At a shared server, the budget belongs per client id, next to the rate limit, and the server can enforce it because it sees every attempt. This site's recommendation: one retrying layer for reads (the server), a per-client retry budget at the gateway, jittered backoff everywhere, and no retries of writes below the host.
Designing the idempotency keyWho generates it, what it covers, how long it lives, and what a conflict means
Who generates it: the caller that knows the intent, which is the agent's application, not the model and not the server. A key generated by the model can change between attempts; a key generated by the server cannot be known by the client before the first reply, which is the reply that gets lost.
What it covers: one intended write. The example's key SUB-1004:run-7f3a:recommendation names the submission, the run and the action, so a new run of the agent writes a new recommendation on purpose, and a retry inside the same run does not. A random key per intent works too; a key derived from content alone is dangerous, because two different runs with the same conclusion would collapse into one row and lose the second run's history.
What it checks: the example stores a hash of the payload with the key. The same key with the same payload is a replay; the same key with a different payload is conflict. Without the hash, a buggy client that reuses a key for a different recommendation would get the old row back and believe its new recommendation was stored.
How long it lives: as long as a retry is plausible, plus the audit window. In the example the key is a column of the recommendation row, so it lives as long as the row, which is the simplest and safest choice for low-volume writes like recommendations. High-volume systems keep keys in a separate table with an expiry.
Where it is enforced: in the domain service, inside the same transaction as the insert, behind a UNIQUE constraint. A check in the MCP adapter ("have I seen this key?") would be a race between two server instances, and it would copy a business rule out of the service, which this site never does.
Where to rate-limit, and by whatGateway or server, per client or per principal or per tenant, and the fairness problems of each
Per client id (the example's choice) matches how teams are funded and how incidents are scoped: the claims assistant has one budget, whoever it acts for. Per principal (the person or agent in sub) protects against one runaway agent instance inside a team, but a team with many users then gets more capacity than a team with few. Per tenant (business unit) protects one unit's capacity from another's but lets one client starve its own unit. Production servers often combine two: a per-client bucket for fairness between teams and a per-principal bucket inside it against runaways.
The gateway is the natural place for coarse limits: it sees every server, can reject before any server work, and can answer HTTP 429. The server is the natural place for limits that need meaning: per tool (writes cheaper to refuse than reads), per tenant (the server knows the submission's unit), or "this call needs the provider". Both layers can enforce limits if they agree on the client id, which both take from the validated token.
One more distinction: a rate limit caps calls over time; a concurrency limit caps calls in flight. A slow dependency hurts through concurrency, because each slow call holds a worker longer while the rate stays the same. A shared server needs both: the bucket against bursts, and a cap on concurrent calls per client so one team's slow calls cannot hold every worker. The example implements only the bucket.
One breaker or many?Per dependency, per tenant or per tool; and why the half-open probe needs care
A breaker per dependency (the example) is right when the dependency fails as a whole. A breaker per tenant or per shard is right when failures are partial: if the commercial-east data lives on a shard that is down, a per-shard breaker keeps commercial-west calls flowing. A breaker per tool fits when tools use different downstream paths: get_assessment_stats reads aggregates and may still work while get_roof_assessment cannot reach the provider.
The breaker's state is also an operational fact that belongs in the records: the example keeps a list of transitions with their times. Every transition should be a metric and an event for the on-call engineer, because an open breaker is an outage that every team sees as fast failures rather than slow ones.
The half-open probe has two traps. If many server instances each have their own breaker, they all probe at the end of the cool-down, and the recovering service gets a burst; jitter the cool-down per instance. And the probe should be a call that tests the dependency, not a write; the example lets any call through, which is fine for a teaching program and risky when the first call after an outage is a write with no key. With idempotency keys on every write, that risk becomes a retry the service can recognize.
Deadlines across hopsThe time left should travel with the request; MCP does not define a field for it
The host gives the whole tool call some time; the server gives its service call 3000 ms; the service gives the provider its own timeout. If each layer sets its own number without knowing the others, an inner layer can keep working long after the outer layer gave up, spending capacity on an answer nobody will read.
The fix used in RPC systems is deadline propagation: the caller sends the time it is willing to wait, and each hop subtracts what it spent and passes the rest on. Normative (MCP specification 2026-07-28): the revision defines request timeouts and cancellation; the pages read here define no _meta key for a remaining deadline. A server can still honor one: on Streamable HTTP, the client's disconnect is the cancellation signal, and the server must treat it as cancellation of that request.
This site's recommendation: propagate the remaining budget in the organization's own _meta key (as this site does with the correlation id), cap it by the server's own deadline, and stop work when the client disconnects. Treat any work that might already have committed, such as a write, as completed-or-not, never as safely undone; the idempotency key is what makes the next attempt safe.
In one breath#
At a shared MCP server the four teams fail together unless the server limits what one failure can take. Give every call a deadline and every attempt a timeout, retry reads a bounded number of times with backoff, never retry writes in the server, and make the client's retry of a write safe with an idempotency key stored UNIQUE with a payload hash. Rate-limit per client id, break the circuit to a dead dependency, tell the model in the error whether a retry can help, and fall back honestly, never by inventing the evidence that failed to arrive.
Each of these behaviors is now in code, and each was shown once, in a scratch run. Once is not a guarantee. The next change to policy.py, a new tool, a second host built against another revision, or a model that follows injected text more eagerly can quietly undo any of them. Turning every promise on this site into a test that fails when it breaks is Testing.