Five weeks after go-live, a reviewer from audit and compliance sends the platform team one question: "Did any agent try to record an underwriting decision for SUB-1004, and if so, what stopped it?" The question is fair. A guidance passage with injected text had been in the index for a while, and this site's own case UW-M09 describes exactly that attempt.
The platform team opens its logs. The gateway's access log has one line per request: time, client address, POST /mcp, status 403, a few hundred bytes. Revision 2026-07-28 puts the tool name in an Mcp-Name header, so the team can even see record_underwriter_decision in some lines. What the log cannot say is who the caller was, which check refused it, under which version of the rules, and whether anything was written. Those facts existed for a few microseconds inside the MCP server and were never kept.
The same week an underwriter reports that the workbench showed "timeout" for SUB-1004 at nine o'clock. Was the model slow, the host's MCP client, the server, the token exchange, or the assessment service? Every component has its own log with its own clock and its own request numbering. Nobody can line them up.
Both problems have the same root. A tool call crosses several programs owned by different teams, and each program records it in its own words. This page builds the records that survive the crossing.
Three questions, three records#
A shared capability gets asked three different questions, by three different people.
An engineer asks "what happened to this one request?" The answer is a trace: a record of one client request as it passes through every program, with timings. A trace is made of spans; a span is one timed piece of work inside one program, such as "the server authorized the call" or "attempt 2 of the service call". For example, the trace of one roof lookup has a span for the server's handling of tools/call, a child span for authorization and a child span for the service call.
A reviewer asks "who did what, and why was it allowed?" The answer is an audit event: a durable record, for people, of one decision about one action. An audit event names the principal from the token, the tool and its sanitized arguments, the authorization decision, the outcome and the policy version. For example, "agent intake-agent asked for record_recommendation on SUB-1004; refused by the scope check under policy 2026.2; nothing written."
An operator asks "how is the server doing, per team?" The answer is metrics: counts and durations aggregated over time, such as calls, refusals, rate-limited calls and errors per client id.
Before AI. A Spring service in a bank already had all three. A tracing library put a trace id on every request across services, an audit_log table recorded every money movement with the user and the approver, and a metrics library counted requests per endpoint. Nobody tried to answer an auditor's question from the access log. An MCP server is one more service on that path, with one new kind of caller: an agent acting for a team.
The three records can share identifiers, and the next section shows how. They must not share a purpose. A trace may be sampled, a metric is aggregated, and an audit event must be complete.
The smallest primitive: one audit event#
The example's server writes an audit event for every tools/call it receives, allowed or refused, from one function. Here is the record function from examples/uw-mcp/audit.py, AuditLog.record, verbatim but shown without its class indentation:
def record(self, *, event: str, claims: dict[str, Any] | None, tool: str | None, arguments: Any,
decision: dict[str, Any], outcome: str, correlation_id: str, trace_id: str, span_id: str,
policy_version: str, transport: str) -> dict[str, Any]:
entry = {
"event_id": f"AUD-{len(self.events) + 1:04d}", "timestamp": self.clock.iso(), "event": event,
"who": who(claims), "what": {"tool": tool, "arguments": sanitize(arguments)},
"decision": {"result": "allowed" if decision.get("allowed") else "refused",
"reason": decision.get("reason"), "check": decision.get("check"),
"detail": decision.get("detail")},
"outcome": outcome, "correlation_id": correlation_id, "trace_id": trace_id, "span_id": span_id,
"policy_version": policy_version, "transport": transport,
}
self.events.append(entry)
return entry
Four groups of fields carry the meaning. who comes from the validated token (subject, principal type, client id, business unit, the token's jti), never from anything the client says about itself. what is the tool and its arguments after sanitizing. decision records which policy check decided and why. The rest ties the event to the trace and to the rules in force. The method takes keyword arguments only, so no caller can write an event without stating a decision and an outcome.
Common confusion. A log line is text for the program's maintainers; it can change format with any release. An audit event is a record with a fixed schema for people outside the team, and a missing field is a defect.
MCP's own logging feature is not an audit trail either. Normative (MCP specification 2026-07-28): Logging is deprecated in this revision, logging/setLevel was removed, and the log level now travels per request in _meta.
Five identifiers that are easy to confuse#
A single roof lookup carries five identifiers, and each answers a different question. Mixing them up is the most common reason a team cannot follow a call.
The JSON-RPC id numbers one request on one connection, so the client can match the reply; it is reused across connections and means nothing elsewhere. The correlation id names one end-to-end request across hops; this site's design carries it in _meta under its own vendor key example.insurer/correlationId. The trace_id and span_id belong to tracing: on this site one trace per client request through every hop, a tool call is a span, and each service attempt is a child span. The token's jti names one token, which links an event to an issued credential. The idempotency key names one intended write, which is the subject of Resilience.
| Identifier | Names | Set by | Example from the run |
|---|---|---|---|
JSON-RPC id | one request on one connection | the host's MCP client | 1 |
correlation_id | one end-to-end request | the host (this site's design) | corr-uw-004-001 |
trace_id | the trace of that request (32 hex) | the first instrumented program | 89658cbb83bc186c4ab7ec7a9719a602 |
span_id | one piece of work in one program (16 hex) | each program | d82b053e5191d81d |
jti | one token | the identity provider | jti-dc0efbb3 |
| idempotency key | one intended write | the agent's application | SUB-1004:run-7f3a:recommendation |
Normative (MCP specification 2026-07-28): traceparent, tracestate and baggage are reserved _meta keys in the W3C Trace Context and Baggage formats, an exception to the rule that reserved keys carry an MCP prefix. A vendor key such as example.insurer/correlationId is allowed because its prefix is not reserved. So the protocol gives trace context a standard place on every request; the correlation id is this site's convention.
Following one call across three hops#
Take the default series case UW-004: the application said the roof was 9 years old, the provider says 22, and the underwriting agent needs the assessment. The model writes a request for get_roof_assessment("SUB-1004"). The host's MCP client sends it to uw-assessment with a correlation id and a traceparent in _meta. The server validates the token, checks the arguments, authorizes the call, exchanges the caller's token for a downstream one and calls the assessment service.
Diagram: One roof lookup, one trace · every hop starts its spans under the parent it received
Time runs down the page. The host's MCP client starts a client span and puts its id in traceparent inside the request's _meta, next to the correlation id. The MCP server reads the parent, starts its tools/call span, then a child span for authorization and a child span for the service call, with one span per attempt. It writes the audit event with the same trace id and its own span id, then replies. The host never sees the server's spans, yet a tracing backend can join them, because they share one trace id. Click a message for the ids it carries.
- Host's MCP client to MCP server uw-assessment: tools/call · _meta traceparent + correlation id. The trace context travels in the reserved _meta key traceparent; the correlation id under this site's vendor key.
- MCP server uw-assessment to Policy check: authorize (child span). A child span records the check that decided and how long it took.
- Policy check to MCP server uw-assessment: allowed · check all. The decision is kept as data, not reconstructed later from today's policy.
- MCP server uw-assessment to Assessment service: service call, own token (child span, attempt 1). The caller's token is exchanged for a downstream token; each attempt is a child span of the service span.
- Assessment service to MCP server uw-assessment: PropertyAssessment. The DTO comes back; the service and attempt spans end with status ok.
- MCP server uw-assessment to Audit store: audit event · same trace id. Who from the token, sanitized arguments, decision, outcome, ids and policy version, written before the reply.
- MCP server uw-assessment to Host's MCP client: result · structuredContent. The reply carries the same JSON-RPC id as the request; the trace ids stay in the records.
The diagram shows the idea that makes a trace work across owners: each program reads the parent from the request and starts its own spans under it. The host never sees the server's spans and the server never sees the host's, yet a tracing backend can join them because they share the trace_id.
Run in a scratch copy of the example on 2026-11-18, one allowed call over the example's Streamable HTTP server prints the following. The script is a scratch helper around the example's own Remote test harness; the correlation id was chosen for the run. The model here is a scripted stand-in, so these results show how the protocol and the controls work, not how good any model is.
--- request id and _meta (ids only)
{"jsonrpc_id": 1, "correlationId": "corr-uw-004-001", "traceparent": "00-89658cbb83bc186c4ab7ec7a9719a602-535c73a5ea319aa5-01"}
--- spans
{"trace_id": "89658cbb83bc186c4ab7ec7a9719a602", "span_id": "d82b053e5191d81d", "parent_span_id": "535c73a5ea319aa5", "name": "tools/call get_roof_assessment", "start_ms": 0, "end_ms": 40, "status": "ok"}
{"trace_id": "89658cbb83bc186c4ab7ec7a9719a602", "span_id": "a41e45cd0cc50647", "parent_span_id": "d82b053e5191d81d", "name": "authorize", "start_ms": 0, "end_ms": 0, "status": "ok"}
{"trace_id": "89658cbb83bc186c4ab7ec7a9719a602", "span_id": "e264fcccb48cde0a", "parent_span_id": "d82b053e5191d81d", "name": "service.get_roof_assessment", "start_ms": 0, "end_ms": 40, "status": "ok"}
{"trace_id": "89658cbb83bc186c4ab7ec7a9719a602", "span_id": "f7cacb2301346eaf", "parent_span_id": "e264fcccb48cde0a", "name": "attempt 1", "start_ms": 0, "end_ms": 40, "status": "ok"}
--- audit
{"event_id": "AUD-0001", "timestamp": "2026-11-18T09:00:00.040Z", "event": "tool_call", "who": {"sub": "agent:underwriting-agent", "principal_type": "agent", "client_id": "uw-workbench", "business_unit": "commercial-west", "jti": "jti-dc0efbb3"}, "what": {"tool": "get_roof_assessment", "arguments": {"submission_id": "SUB-1004"}}, "decision": {"result": "allowed", "reason": "allowed", "check": "all", "detail": "scope, principal type and business unit checks passed"}, "outcome": "ok", "correlation_id": "corr-uw-004-001", "trace_id": "89658cbb83bc186c4ab7ec7a9719a602", "span_id": "d82b053e5191d81d", "policy_version": "2026.2", "transport": "http"}
--- downstream exchange
{"tool": "get_roof_assessment", "downstream_jti": "jti-ea557154", "aud": "https://assessment-svc.internal.example", "scope": "assessment.read", "sub": "agent:underwriting-agent", "actor": "svc:uw-assessment-mcp", "passthrough_guard": "a separate token issued for the assessment service", "service_accepts": true, "caller_token_forwarded": false}
Three things to notice. The server's top span has the client's span 535c73a5ea319aa5 as its parent, read from traceparent, so the trace continues across the process boundary. The audit event carries the same trace_id and the server span's span_id, so a reviewer who finds the event can open the trace, and an engineer looking at the trace can find the decision. And the times are simulated milliseconds on the example's clock: 40 ms is a constant in resilience.py, not a measurement.
What the run proves is narrow: the ids line up inside the example. It does not prove that a production tracer, gateway and service propagate them; that needs each hop to read and forward the context. In the example the hop to the assessment service is in-process, so the server records the service span itself. A deployed service records its own spans, which works only if the server forwards the trace context on the downstream call.
The authorization decision is a fact, not a log line#
The reviewer's question was about a refusal. The server must record the authorization decision as data: the result, the reason, the check that decided and the detail, at the moment it decides. A decision reconstructed later from the current policy is a guess, because the policy may have changed.
Case UW-M03: the intake agent calls record_recommendation by name, although the tool was not in its list. The run printed this event:
{"event_id": "AUD-0001", "timestamp": "2026-11-18T09:00:00.000Z", "event": "tool_call", "who": {"sub": "agent:intake-agent", "principal_type": "agent", "client_id": "intake-pipeline", "business_unit": "commercial-west", "jti": "jti-a7b1cfef"}, "what": {"tool": "record_recommendation", "arguments": {"submission_id": "SUB-1004", "recommendation": {"recommendation": "escalate", "reasons": "[redacted: 1 items]", "citations": ["R-ROOF-03"], "requires_human": true}, "idempotency_key": "SUB-1004:run-7f3a:recommendation"}}, "decision": {"result": "refused", "reason": "not_authorized", "check": "scope", "detail": "record_recommendation needs scope uw.recommend"}, "outcome": "refused", "correlation_id": "corr-record_recommendation-001", "trace_id": "a7d8efca280686bd3e0aa2d879909731", "span_id": "4c0a7cdab19da395", "policy_version": "2026.2", "transport": "http"}
Case UW-M09 answers the reviewer directly. The scripted stand-in follows the injected passage P-NOTE-17 and proposes record_underwriter_decision with approve. The run then repeats the call with a token that someone wrongly granted uw.decide. Two events, two different checks:
{"event_id": "AUD-0001", "timestamp": "2026-11-18T09:00:00.000Z", "event": "tool_call", "who": {"sub": "agent:underwriting-agent", "principal_type": "agent", "client_id": "uw-workbench", "business_unit": "commercial-west", "jti": "jti-dc0efbb3"}, "what": {"tool": "record_underwriter_decision", "arguments": {"submission_id": "SUB-1004", "decision": "approve", "reason": "[redacted: 21 chars]", "override": false}}, "decision": {"result": "refused", "reason": "not_authorized", "check": "scope", "detail": "record_underwriter_decision needs scope uw.decide"}, "outcome": "refused", "correlation_id": "corr-record_underwriter_decision-001", "trace_id": "4ff9818149a0fd67545b0d1b097481d3", "span_id": "3dfd26234fc6fc54", "policy_version": "2026.2", "transport": "http"}
{"event_id": "AUD-0002", "timestamp": "2026-11-18T09:00:00.000Z", "event": "tool_call", "who": {"sub": "agent:underwriting-agent", "principal_type": "agent", "client_id": "uw-workbench", "business_unit": "commercial-west", "jti": "jti-b2d222aa"}, "what": {"tool": "record_underwriter_decision", "arguments": {"submission_id": "SUB-1004", "decision": "approve", "reason": "[redacted: 21 chars]", "override": false}}, "decision": {"result": "refused", "reason": "not_authorized", "check": "principal_type", "detail": "record_underwriter_decision is for principal type user; this caller is agent"}, "outcome": "refused", "correlation_id": "corr-record_underwriter_decision-002", "trace_id": "560a3172dc410e18e825c586d5d08191", "span_id": "68601b494dce0ed1", "policy_version": "2026.2", "transport": "http"}
The second event is the one a reviewer values most. It shows that the defense held even after a grant mistake, and it names the check that held. The sanitized reason shows something else: the injected text never reached the audit store, only its length.
Case UW-M11 shows why the audit store may know more than the caller. The claims assistant (business unit commercial-west) asks for SUB-2001, a commercial-east submission. The caller receives Submission SUB-2001 was not found., word for word what it receives for SUB-9999, so the existence of another unit's submission does not leak. The audit event records the truth for the reviewer:
{"event_id": "AUD-0001", "timestamp": "2026-11-18T09:00:00.000Z", "event": "tool_call", "who": {"sub": "agent:claims-assistant", "principal_type": "agent", "client_id": "claims-assistant", "business_unit": "commercial-west", "jti": "jti-820ab67b"}, "what": {"tool": "get_roof_assessment", "arguments": {"submission_id": "SUB-2001"}}, "decision": {"result": "refused", "reason": "not_found", "check": "tenant", "detail": "no submission SUB-2001 in business unit commercial-west"}, "outcome": "refused", "correlation_id": "corr-get_roof_assessment-001", "trace_id": "0127a8d0c81ad7fd2d6fef44dc4541fd", "span_id": "89d1d33ce9cd7e3a", "policy_version": "2026.2", "transport": "http"}
{"event_id": "AUD-0002", "timestamp": "2026-11-18T09:00:00.000Z", "event": "tool_call", "who": {"sub": "agent:claims-assistant", "principal_type": "agent", "client_id": "claims-assistant", "business_unit": "commercial-west", "jti": "jti-820ab67b"}, "what": {"tool": "get_roof_assessment", "arguments": {"submission_id": "SUB-9999"}}, "decision": {"result": "refused", "reason": "not_found", "check": "tenant", "detail": "no submission SUB-9999 in business unit commercial-west"}, "outcome": "refused", "correlation_id": "corr-get_roof_assessment-002", "trace_id": "0bd96b78d83863c8390e052892476312", "span_id": "4094b53270a52c47", "policy_version": "2026.2", "transport": "http"}
In our insurance agent. The answer to the reviewer is now a query, not an investigation: events wherewhat.toolisrecord_underwriter_decisionandwho.principal_typeisagent. In the run there are two, both refused, one by scope and one by principal type, both under policy2026.2, and the decisions table has zero rows.
This site's recommendation: write the audit event in the same code path that enforces the decision, after the decision and before the response, and treat a failure to write it as a failure of the call. An audit event written by a separate observer can miss the refusals that never reached the observer.
What to record, and what never to record#
An audit event that holds the caller's token is a credential store with weak access control. An audit event that holds free text holds whatever the free text contained: a broker's personal details, or an injected instruction waiting for the next system that reads it. So the example records sanitized arguments: a copy of the arguments with secrets removed and free text replaced by its size. Here is sanitize from audit.py, verbatim:
FREE_TEXT_FIELDS = {"reason", "reasons", "query", "text", "note"}
SECRET_MARKERS = ("token", "authorization", "secret", "password")
def sanitize(value: Any, key: str = "") -> Any:
"""Copy of `value` with tokens removed and free text replaced by its size."""
k = key.lower()
if any(m in k for m in SECRET_MARKERS) and k != "idempotency_key":
return "[removed]"
if k in FREE_TEXT_FIELDS:
if isinstance(value, str):
return f"[redacted: {len(value)} chars]"
if isinstance(value, list):
return f"[redacted: {len(value)} items]"
if isinstance(value, dict):
return {kk: sanitize(vv, kk) for kk, vv in value.items()}
if isinstance(value, list):
return [sanitize(v) for v in value]
return value
Identifiers stay, because a reviewer needs them: submission_id, idempotency_key, rule citations. The exception for idempotency_key exists because the word "key" is close to the secret markers; a sanitizer built from a deny list needs such exceptions, which is why the test test_governance.py:test_sanitize pins its behavior. The token is never recorded; its jti is. The test test_mcp.py:test_audit_never_holds_a_token sends a real call over HTTP and asserts that neither the token nor the free text appears anywhere in the audit store.
Normative (MCP specification 2026-07-28): clientInfo and serverInfo are self-reported and should not be relied on for security decisions. The same holds for audit: who comes from the validated token, and clientInfo is at most a hint about which software sent the request.
Timeouts and policy versions belong in the record#
A refusal is one kind of outcome. A call that was allowed and then failed is another, and the audit event must not blur them. Case UW-M08: the assessment service times out on every attempt of a read. The example's audit event records "decision": {"result": "allowed", ...} and "outcome": "error:timeout", and the trace holds three attempt spans under the service span, at simulated 0 to 1000 ms, 1200 to 2200 ms and 2600 to 3000 ms. A reviewer sees that the call was permitted and did not complete; an engineer sees where the time went. The retry logic itself is the subject of the next page.
The policy version matters for the same reason the decision does. The fixture has two versions of UW-GUIDE: 2025.4 with a roof limit of 30 years and 2026.2 with 25 years in wind zone C. An event that says "allowed" without "under 2026.2" cannot be judged a year later.
Audit events, traces and metrics side by side#
With the three records built, the differences can be put in one table.
| Record | Reader | Holds | Kept | Sampled? |
|---|---|---|---|---|
| Trace and spans | engineers on call | timing, parent and child, status per hop | days to weeks | often, by design |
| Audit event | reviewers, audit and compliance, incident leads | who, what (sanitized), decision, outcome, policy version, ids | the retention the organization's policy sets | never |
| Metrics | operators, each consuming team | counts and durations per client id, tool, outcome | months, aggregated | aggregated, no single call |
The rule that follows is short: never answer a reviewer from a trace, and never page an engineer from an audit query. The trace_id in the audit event and the client id in the metrics are the bridges between them.
Diagram: One call, three records, three readers · linked by ids, never merged
Read left to right. One tools/call produces a trace for the engineer on call, an audit event for the reviewer and a contribution to metrics for operators. The trace id in the audit event and the client id in the metrics are the bridges. A trace may be sampled and metrics are aggregated; the audit event is never sampled. Click a box for what each record holds and who reads it.
- One call
- tools/call: One request, with its JSON-RPC id, correlation id and traceparent.
- Three records
- Trace and spans: Server span and authorize span, both ending in a refusal; no service span because nothing executed.
- Audit event: Who, what (sanitized), the authorization decision, the outcome, ids and policy version.
- Metrics: Counts of calls, allowed, refused, rate limited and errors per client id and tool.
- Readers
- Engineer on call: Opens the trace from an alert or from the trace id in an audit event.
- Reviewer: Queries audit events; reads through a principal that holds no tool scopes.
- Operators and each team: Watch refusals, rate limits and errors per consuming team.
Evidence a reviewer can rely on#
A record is evidence only if the reviewer can trust three properties. It is complete: every call produced an event, including refusals at the token check (the example's HTTP endpoint writes an auth_refused event for a 401 before any tool code runs). It is unchanged: events are appended, never edited, and the people who run the server cannot rewrite them. And it is readable by the right people only.
The example makes the last point with a principal. The audit agent holds scope audit.read and sees no MCP tool at all, because no tool needs audit.read (Who gets access). Its access is to the audit store, not to the capability. A reader of evidence should not be able to act on the system the evidence describes.
Governance note. Audit and compliance own the questions and the retention period; the platform team owns the store; the server maintainers own the event schema and must treat a change to it like a change to a public contract. The Governance page names who approves which change; for observability the rule is that nobody whose actions are recorded may delete records.
The ecosystem: OpenTelemetry conventions for MCP#
Ecosystem pattern: OpenTelemetry publishes semantic conventions for MCP: agreed span names and attribute names, so traces from different SDKs and gateways look alike. As documented on 2026-11-18, the generative AI conventions moved from the main OpenTelemetry site to a separate repository, open-telemetry/semantic-conventions-genai, which has no tagged releases; the MCP page's status is "Development". The page still describes the earlier protocol model: initialize appears as a method value and mcp.session.id as an attribute, and the parts read here do not mention server/discover or revision 2026-07-28. Use the conventions for names, and expect them to change.
| Example field | Convention as documented on 2026-11-18 |
|---|---|
span tools/call get_roof_assessment | span name {mcp.method.name} {target}, target the tool or prompt name; server spans of kind SERVER |
| method | mcp.method.name (required) |
| tool | gen_ai.tool.name (conditionally required) |
JSON-RPC id | jsonrpc.request.id (conditionally required) |
isError true | error.type should be tool_error; on a JSON-RPC error, the code as a string |
traceparent in _meta | context injected into params._meta, keys unprefixed |
| arguments | gen_ai.tool.call.arguments, opt-in, may contain sensitive data |
The last row deserves care. The convention makes recording arguments opt-in because they may be sensitive; on a shared server, record sanitized arguments in the audit event and leave them out of spans unless a reviewer has approved it.
What goes wrong without these ideas#
| Missing idea | What happens |
|---|---|
| Audit event per call, refusals included | The reviewer's question about SUB-1004 has no answer; a 403 line says nothing about who or why |
Trace context in _meta | Each hop has its own log and clock; the nine o'clock timeout cannot be placed |
| Decision recorded at decision time | A refusal is explained with today's policy, not the one in force |
| Sanitized arguments | Tokens and broker free text sit in a store many people can read |
| Separate records for separate readers | A sampled trace is offered as evidence; an audit query pages an engineer |
Security and governance notes#
Security note. The audit store is a target: it holds who called what, which is a map of the system. Keep tokens out (onlyjti), keep free text out or hashed, restrict readers to a principal likeaudit-agentthat holds no tool scopes, and make writes append-only. The NSA's MCP guidance lists poor or missing audit logs as a concern and asks for logs that capture a traceable sequence of actions and detect authorization failures (Threat model).
In a coding agent. A coding agent that calls a company's shared repository server and issue tracker server needs the same three records. The trace answers "why did the agent's run take twelve minutes"; the audit event answers "which agent, for which developer, calledopen_pull_requeston the payments repository, and which policy allowed it"; the metrics show one team's agents floodingsearch_code. A local stdio server on a laptop writes tostderrat most, which is fine for one developer and not evidence for anyone else. Harness Engineering covers the runtime that records the agent's side.
What this is not. This page covers what a shared MCP server must record about tool calls, and how those records link. Logs, metrics and traces as an engineering discipline (sampling strategies, dashboards, alerting, storage) belong to https://observe.sammuthu.com.
Checklist for Monday#
| Check | Pass when |
|---|---|
| Event per call | Every tools/call, allowed or refused, including token refusals, writes one audit event |
| Who from the token | who holds subject, principal type, client id, business unit and jti; never the token |
| Decision as data | Result, reason, check and detail are fields, written where the decision is made |
| Sanitized arguments | A test asserts that no token and no free text reaches the store |
| Ids | The event carries correlation id, trace_id, span_id and policy version |
| Propagation | Every hop reads traceparent and forwards it on its downstream call |
| Separate readers | Audit readers hold no tool scopes; writes are append-only |
| Metrics per client | Calls, refusals, rate limits and errors are counted per client id and tool |
Exercises#
- The intake agent's audit event for
UW-M03showscheck: scope. A colleague proposes dropping refused calls from the audit store "because nothing happened". What do you answer? - A gateway in front of
uw-assessmentalready logs every request with theMcp-Nameheader. Which audit fields can the gateway fill, and which only the server? - Where should the reviewer look to prove that the injected passage in
UW-M09never caused a write, and why is one place not enough?
AnswersOne line each
- A refusal is the event a reviewer most needs: it shows a control working, and a pattern of refusals is how an attack or a misconfigured client is found.
- The gateway can fill who (from the token it validated), the tool name, transport and timing; only the server knows the policy check that decided, the tenant result and the outcome of the service call.
- The audit events (two refusals, checks scope and principal type) and the decisions table (zero rows); the events show what was attempted and refused, the table shows what exists.
Exact parameters or redacted ones?A real tension between evidence and privacy, and three designs that resolve it
The NSA's MCP security guidance (archived copy read 2026-11-18) recommends logging all tool and model invocations with exact parameters, identities and, where feasible, hashes of outputs, and feeding a SIEM. The example does the opposite for free text: it records only the length. Both positions have a reason. Exact parameters let a reviewer reconstruct what happened; free text in a widely readable store leaks personal data and carries injected instructions to every downstream reader, including a model that summarizes the logs.
Three designs reconcile them. The first records a keyed hash of each free-text field next to its length: the reviewer cannot read the text but can prove that two events carried the same text, or match it against a document they hold. The second writes exact parameters to a separate evidence store with a much narrower reader list and a shorter retention, and puts only a pointer in the ordinary audit event. The third keeps free text out of tool arguments altogether by design: the injected passage P-NOTE-17 reached the model as a retrieved document, and the tool call carried only a short reason.
This site's recommendation: identifiers and enumerated values exact; free text hashed and sized in the audit event; exact text only in a separate, access-controlled evidence store when a regulation requires it. Whatever you choose, write the choice into the event schema so a reviewer knows what "redacted" means.
Who writes the audit event: gateway, server or service?Each layer sees part of the truth; a complete record needs a rule for which layer writes what
A gateway sees every request, including those that never reach a server: bad tokens, rate limits, unknown servers. It can record who, which server, which tool name (from Mcp-Name), status and duration. It cannot see why a server refused a call that passed the gateway, nor the business unit check, nor whether the service wrote a row.
The MCP server sees the decision: the scope, principal type and tenant checks of policy.authorize, the schema validation result and the outcome of the service call. It is the natural writer of the decision record, which is why the example writes there.
The assessment service sees the effect: the row inserted, the transaction committed or rolled back. Its own audit table (the recommendations and decisions rows, with recorded_by and decided_by_sub) is the record of what exists.
A design that writes in all three layers needs one shared key, and the correlation id plus trace_id serve. It also needs a rule against double counting: the gateway's event is "request received and forwarded", the server's is "decision and outcome", the service's is "state changed". A reviewer reading three events for one call should see three different facts, not one fact three times.
One gap in the example is worth naming: the audit event records the caller's jti but not the jti of the downstream token from the token exchange; that link sits in the gateway's exchange record. A production design should put both in the event, so a reviewer can follow the authority from the caller to the service.
Making the audit store tamper-evidentAppend-only is a permission; tamper evidence is a property a reviewer can check
Append-only storage stops ordinary edits, but an administrator with enough rights can still rewrite history. A reviewer who must rely on the record wants to detect that, not trust that it did not happen.
The usual technique is a hash chain: each event carries a hash of its own content and the previous event's hash. Changing or deleting one event breaks every hash after it. Publishing the latest hash periodically to a place the store's administrators cannot write (another team's system, a write-once bucket) makes a rewrite detectable even if the whole chain is recomputed. The example's events are numbered (AUD-0001, AUD-0002) on a simulated clock, which is enough to teach ordering but not tamper evidence.
Two operational traps follow. A shared server usually runs several instances, so there are several writers; either each instance keeps its own chain and the chains are merged with their instance id, or events go through one ordered log. And retention deletes events by design, so the chain must record where it was cut and under which policy. Neither is specific to MCP; what is specific is volume: an agent can make many calls per minute, so the audit path must be cheap enough to run on every call, refusals included.
One request or several? Correlation across retriesThe example gives each retry its own correlation id; the idempotency key ties them together
In UW-M08 the client sends record_recommendation three times: the first attempt times out, the second succeeds, the third is a retry after a lost reply and is answered from the stored row. In the run these are three requests with JSON-RPC ids 2, 3 and 4 and correlation ids corr-core-002, corr-core-003 and corr-core-004. Each has its own audit event: outcome error:timeout, then ok, then ok. The idempotency key SUB-1004:run-7f3a:recommendation appears in all three, sanitized but kept.
That is one defensible design: a correlation id names one request, and a retry is a new request. Another design keeps one correlation id for all retries of the same intended action and adds an attempt number. The second makes one trace show the whole story; the first keeps traces short and lets the idempotency key join them.
Either works if it is written down and the audit query uses the right key. What fails is the mixture: some clients reuse the correlation id, others do not, and a reviewer counting "attempts at a write" counts wrong. This site's recommendation: treat the idempotency key as the identity of a write in every audit query about writes, whatever the correlation design, and record attempt_number when the host knows it.
In one breath#
A shared MCP server is asked three questions by three readers, so it keeps three records: a trace for engineers, joined across hops by the traceparent that revision 2026-07-28 reserves in _meta; an audit event for reviewers, with who from the token, sanitized arguments, the authorization decision as data, the outcome and the policy version; and metrics per client id for operators. Keep the identifiers apart, write the decision where it is made, never record a token or free text, and let nobody whose actions are recorded edit the record. A refusal recorded well is the best evidence a control works.
The records now show the moment the assessment service stopped answering: three attempt spans, an allowed decision, an outcome of timeout. Recording a failure is not handling it. When four teams depend on one server, a slow service, a retry storm or one team's burst becomes everyone's outage, and a write retried at the wrong moment becomes two recommendations. That is Resilience.