Self-funded research by IsoKron, conducted and written with AI assistance.
An MCP audit is a measurement, and a measurement is only as good as its instrument. At IsoKron, the harness that runs the agents, the proxy that records their traffic and the scripts that grade their work go through the same adversarial review as the findings they produce. Nothing is filed upstream until both have held up.
This article shows how that works, using our audit of GitHub's official MCP server v2.1.0 as the worked example. It covers the rules we run under, the defects that review caught in our own tooling and what we changed, the refutation funnel every claim passes through, and a checklist you can use to test your own MCP server.
The problem: an agent audit is only as good as its measurement
In an agent audit, the auditor is software too. A harness launches the agents, a proxy records what they send and receive, a script grades their answers and a sweep counts tokens. Each of those can be wrong in a way that looks exactly like a finding about the server under test. Agent evals share the weakness: a score is only as trustworthy as the harness and grader behind it.
Four risks matter most in MCP server testing with agents:
- Harness bugs. A counter that measures the wrong fields, or a fetch step that saves a proxy's error page in place of real data, produces confident numbers about nothing.
- Answer-key leakage. If an agent can read the expected answer, an accuracy score measures reading, not the server.
- Graders. A grader that pattern-matches text scores formatting, not correctness. In our read-only matrix of 27 runs, a regex grader marked 6 correct answers as misses because they were written as tables or as a bold "No."
- Stale tools and stale facts. Local dependencies drift, the target's tool catalog changes between builds, and documentation disagrees with what a build actually serves. Our pre-audit research put GitHub's server at 92 tools in total and 45 by default; the pinned v2.1.0 build served 91 and 46.
During the GitHub audit, review of our own toolchain found a fetch step that saved a proxy error page as data, an answer key stored where operator agents could read it, a regex grader that scored correct answers as misses, and documentation that disagreed with the build's tool count. Each was caught before a number reached a client or an upstream issue. The rest of this article is how.
Our rules
Six rules govern every IsoKron audit run. Each one closes a specific way for the instrument to fool us.
1. A frozen, hashed harness
Before the first measured run, every harness file is hashed and the hash list is recorded. For the before/after benchmark, the run and grading scripts were hashed and checked before the first run. We fell short of the rule in one respect: file timestamps show that the analysis script, which only post-processes captured results, and the hash list were last written about two minutes after the first run began. The final list (sha256 a08ebac0…) was checked at the start of the second phase, and the only parameter that changed between phases was the server image.
The harness changes only between phases, never mid-run, and a change gets a new hash. When the write phase needed a write-scoped token, the capture proxy gained a token-file setting and was re-hashed (656d5edf…) before any write run started. Every run ID records its phase, and the benchmark report records the server image digest for each phase.
2. Raw JSON-RPC capture through a tee
Agents never talk to the server directly. A small stdio proxy, the tee, sits between client and server, forwards every byte unchanged, and logs each JSON-RPC line with a timestamp, its direction and its size. Call counts, wire bytes and error rates come from that log, not from the agent's own account of what it did. The tee redacts the access token at capture time, before anything reaches disk; since the K-11 fix below, it also strips the tokens in signed URLs.
3. A clean-context refuter on a different model
A finding is a claim until a second agent fails to break it. Our finders were Claude Opus developer runs and scripted probes. Our refuters were Claude Sonnet, each started with a clean context and only the claim, writing its own reproduction and checking the server's behaviour against the underlying REST or GraphQL API as a control. A refuter returns one of four verdicts: confirmed, narrowed to the conditions under which the claim holds, refuted, or already reported upstream.
4. Deterministic probes with no model in the loop
Concurrency, retries, partial failures and malformed parameters are tested with scripted JSON-RPC, not with an agent. A model adds variance where we need a controlled experiment. The write-side probe track made 261 writes, and each of its findings was re-run once on a clean fixture by the finder before it went to a refuter.
5. Retest before filing
Before anything goes upstream it is re-run on the audited release and on the latest main-branch build, and searched against existing issues. For GitHub's server, every item was re-run on v2.1.0 and on the main-branch nightly (commit 99d6d650). All reproduced on both. The duplicate search found that one finding, a README inconsistency, was already reported with a fix pull request open, so we did not file it.
6. The dual-track rule
Audit agents run two tracks. The target track is the client's server. The toolkit track records defects in our own instruments, with a note on whether each one blocks the audit. Toolkit work is capped at a fixed share of each agent's token budget: enough to record and work around a defect, not enough to turn an audit into a tooling project. Fixes land in the tool's own repository and release. During the GitHub audit the toolkit track logged 18 items, two of them marked blocking.
What reviewing our own instrumentation caught
The toolkit track is where defects in our own MCP audit tools are recorded, and the review of our instrumentation is what fills it. These four entries from the GitHub audit are the most instructive, because each one produced output that looked plausible.
| Item | Defect | Effect | Fix |
|---|---|---|---|
| K-1, catalog sweep | The static sweep counted tool icons (base64 image data in tools/list) as model exposure. | It reported 119,633 cl100k tokens for 91 tools when the model-visible text is about 23.4K: a 5x overstatement. Icons were 145,472 bytes, 56% of the response. | The sweep now reports two separate measures: model-visible tokens (name, description, input schema) and wire bytes (the whole entry). Budgets apply to model-visible tokens only. |
| K-10, grader | A regex fact-grader matched the wording of answers, not their content. | 6 of 27 correct answers were scored as misses because they used a table or a bold "No." | Task success is no longer scored from answer text. The benchmark scores each run with deterministic end-state checks: a separate read-only checker reads the repository through the GitHub API and tests what the task changed. |
| K-11, capture | The tee stored signed raw-file download URLs returned by the target for its private repository. | 40 capture files held short-lived, reusable access URLs. The defect blocked publishing. | All 40 files were redacted, and the tee now strips token= values at capture time, before anything reaches disk. |
| K-13, fixture guard | The protected-fixture check compared only each issue's updated_at. | Probe edits that mentioned issue numbers added 20 cross-reference events to protected issues 1 to 10 without changing updated_at, so the guard passed. | The guard now checks the timelines of protected issues as well as their fields. |
Two patterns run through the list. First, K-1 and K-10 both made the target look worse than it was: a 5x token figure, and correct answers scored as misses. Second, each fix is structural. It changes what the tool measures, what it records or how success is scored, so the same mistake cannot recur quietly. None of these defects changed a finding we filed: affected figures were recomputed or checked by hand, and on manual review every read-only run had found the expected facts.
The refutation funnel
Every claim from the GitHub audit passed through the same gates, and at each gate some claims died or shrank. On the write side, refuters confirmed 7 of 14 claims as stated, narrowed 6 and refuted 1.
| Gate | Read side (static pass) | Write side (probes and developer runs) |
|---|---|---|
| Carried to a verdict | 13 candidates | 14 claims |
| Confirmed as stated | 4 | 7 |
| Narrowed to the conditions where they hold | 4 | 6 |
| Refuted | 1 | 1 |
| Already reported upstream | 4 | 3 (also counted above) |
On the read side, 8 candidates survived as new findings. 6 of the 8 were confirmed against the live server; G-6, a Projects field gap, rests on source and API documentation only.
Survivors then went through the retest gate: re-run on v2.1.0 and on the main-branch nightly, plus a duplicate search. Everything re-run reproduced on both builds, and the duplicate search removed one finding that already had an upstream fix in progress. What remained became 11 new upstream issues on github/github-mcp-server, plus reproductions added to 2 existing issues.
The funnel changed what we said, not only how much:
- Labels. The finder saw 6 of 11 concurrent label adds lose one label and called it a race. The refuter showed that truly simultaneous adds lose about 1 in 5, and that the reliable loss is an agent reading the labels and writing the full set back. The finding was restated that way.
- Issue fields. The finder's baseline was the previous pair's read-back. The refuter used a fresh GraphQL baseline for every pair and still saw 4 of 8 pairs lose a change, against 0 of 6 for the per-field control. Across all three runs: 19 of 30 lost, 0 of 28 in the control.
- Empty code search. Agents got zero results from
search_codeon the private repository. The refuter got the identical empty, incomplete result from GitHub's REST API, so this is GitHub's indexing, not the MCP server. Refuted as an MCP bug. - Signed URLs. A draft issue said the server strips two URL fields from directory listings. The recheck before filing showed it does not, and the issue was corrected before it went out.
- Tool totals. A claim that the README miscounts the server's tools was refuted as worded: the README lists 92 names, which is the 91 served tools plus
delete_repository. - Our own catalog-size note. An early internal note, written after the first eager-loading run, called tool-catalog size a cost problem. That run was a cold-cache outlier, and the full set of 18 eager-loading runs did not support the note. It was corrected before anything was published.
We apply the same funnel to our own conclusions as to claims about the target.
Tamper-evident evidence
A frozen harness only helps if someone else can check that it was frozen before the runs, not after. kronicle-lite, IsoKron's open-source hash-chained store, is built for that. Each artifact and each milestone (harness frozen, every run completed, report issued) is recorded as an event on a SHA-256 chain that anyone can recompute from a public starting point.
Signed checkpoints of that chain are countersigned by IsoKron's anchor service and timestamped by independent RFC 3161 timestamp authorities. A later rewrite of an anchored range is detectable, and the order "harness frozen, then runs, then report" can be checked offline by a third party with one command. The evidence establishes integrity, order, and that each checkpoint existed no later than its timestamp; the depth and method of an audit are set out in its report.
Our next article walks through that verification end to end.
How to test your own MCP server: a checklist
The same rules apply to a team testing its own MCP server before a release. None of them needs more than an agent client, a logging proxy and a throwaway environment.
- Pin what you test. Record the version, commit and image digest. Snapshot
tools/listand fail CI when it changes; mcp-catalog-snap does this with a sha256 fingerprint and a readable diff. A changed description or schema changes agent behaviour even when no code path did. - Count tokens and bytes separately. Report model-visible tokens (name, description, input schema) apart from wire bytes. Icons, output schemas and metadata cross the wire but usually never reach the model.
- Use a sealed target, and guard it. A throwaway organization or tenant, fake data, a planted canary secret, and least-privilege tokens with a short expiry. Use separate read and write tokens, and check that each fails where it should. Check that protected objects are unchanged, timelines included, not just modification times, and reseed rather than reuse a fixture that a probe has touched.
- Capture raw JSON-RPC. Put a logging proxy between client and server and compute every number from its log. Redact tokens and signed URLs at capture time, and scan captures for secrets before you share them.
- Freeze and hash the harness. Hash every harness file before the first measured run. Change one variable between phases, and change the harness only between phases.
- Probe write paths without a model. Script the hard cases: two server processes writing to the same object, stale reads, retries after a dropped connection, partial failures, unknown parameters, and annotations that disagree with behaviour. Run the same operation through the underlying API as a control.
- Run real agent tasks for usability. Count calls, turns and failed calls per task, at least 3 runs each. Report ranges. Overlapping ranges show no difference; at 3 runs, non-overlapping ranges are a screen, not a significance test.
- Grade end state, and keep the answer key out of reach. Check what the task changed through a separate read-only pass. If you grade answers, use a structured answer format and audit every miss by hand. Store expected results outside every path an agent, or an operator agent, can read.
- Refute, retest and search before filing. Hand each claim to a second agent on a different model, with a clean context and nothing but the claim; expect narrowed claims, which are usually the accurate ones. Re-run survivors on the release you tested and on the latest main build, and search existing issues. A finding that is already reported gets a reproduction comment, not a new issue.
- Log your own defects. Keep a toolkit track next to the findings, cap the time it takes, and fix each defect in the tool, not in a one-off script.
About IsoKron audits
IsoKron audits MCP servers for the teams that build and deploy them, and every audit runs under the rules above. The process uses three open-source tools IsoKron maintains: mcp-catalog-snap for catalog snapshots and sweeps, kronicle-lite for the evidence chain, and sea-lite for tracking each finding through its gates.
For the findings this process produced on GitHub's server, read our audit of GitHub's MCP server v2.1.0. To have your own server audited, see MCP server audits.