Skip to content
IsoKron

Method

How we test and verify

Every finding in a client report passes three checks before it reaches you, and every benchmark runs on a harness frozen before the first measured run. This page sets out both, with a finding we dropped.

How every finding is verified

  1. Step 1: Checked against a control

    A script reproduces the finding, and the same operation runs through the vendor’s own API where one exists, so we know whether the bug is in the MCP server or the platform.
  2. Step 2: Challenged by a separate AI refuter

    A separate AI, in a clean context and often on another model, tries to explain the finding away. If it can, the finding is dropped.
  3. Step 3: Reproduced and reviewed by a person

    A person reruns the reproduction and reads the evidence before anything reaches the report.

This is the standard for client reports. Our public audits say which of these checks their findings went through, and where a check fell short.

A finding we dropped

Evidence:

In the GitHub audit, search_code returned nothing on a private repository. The refuter ran the same search through GitHub’s REST API: also nothing. A GitHub indexing effect, not an MCP bug, so it was never reported.

How we benchmark

Benchmark harnesses are hashed and frozen before the first measured run, calls are throttled, and every number has a denominator and a control.

A rebuild is measured on two task sets: tasks frozen before the changes are designed, and tasks drawn from the friction we find, which favor the rebuild by construction. Each set is reported on its own, whatever it shows, and no gain is promised.

The research builds on the build page were tested on our own test setups, with one AI agent per run. Three runs per task show spread, not significance. We report a client’s before/after the same way, whatever it shows.

How to test an MCP server without fooling yourself

Terms, in plain words

Write path
The tools that change data (create, update, delete), as opposed to the ones that only read.
tools/list (the catalog)
The list of tools a server offers to agents, with their descriptions and input schemas.
Annotation honesty
Whether the hints a tool declares about itself (read-only, destructive) match what it actually does.
Test tenant
A separate account or workspace on your platform with test data only, so testing never touches production.
Evidence Pack
The signed, timestamped record of an audit, which you can check on your own machine with one command. How to verify one.
Research build
Our own changes to a vendor’s open-source MCP server, tested and published: not a client job and not an official release.

Where this applies: MCP server audits and MCP server builds.

Request an audit or a build

Tell us which server and what you need. We reply to confirm the scope, the price and the dates.