Self-funded research by IsoKron, conducted and written with AI assistance. Not commissioned or endorsed by Payload; IsoKron is not affiliated with Payload. IsoKron sells MCP server audits and builds.
An editor, editor-b, has a post open in Payload’s admin panel, so Payload holds a document lock for her. An agent is asked to add a row, Loppers, to the post’s list, and reads the post. Just after that read, editor-b saves a row of her own, Wheelbarrow, and keeps editing. The agent then writes back the list it read, plus Loppers, through updatePosts, the update tool of Payload’s official MCP plugin, @payloadcms/plugin-mcp 3.93.0. Payload’s REST API refuses such a write with HTTP 423 while another user holds the lock. The MCP tool let it through, deleted editor-b’s lock row and replaced the list, so Wheelbarrow was gone, and the agent told the user it was done.
Two things combined there. The row was lost because the agent wrote from a read taken before her save, and no Payload surface checks for that. The lock is what should have stopped the write: REST enforces it, while the plugin inherits the default of Payload’s Local API, which lets trusted server code write through locks. Her admin-panel session was simulated: our harness held her lock the way the admin panel does and made her save through REST, so what her open form would do next was not tested. That was one native session run; on the vendor build the same task then failed 5 of 5 benchmark runs.
The control arm, the same change sent straight to Payload’s REST API, tells the plugin’s defects apart from the platform’s. If REST loses nothing, the MCP layer causes the loss; if REST loses the change too, the platform has the problem, and the MCP layer can still add to it. The lock bypass is in the plugin: REST enforces the lock. A stale write, and two simultaneous partial updates to different fields, lose changes through REST as well, so those are Payload’s platform behavior; we report them so the plugin’s findings are not overstated.
We audited Payload’s plugin on 13 November 2026: scripted probes against a pinned build with REST controls, refuter agents, a native agent session, a research build with every change, and a 144-run before/after benchmark. A completeness pass the same evening re-ran the first ten findings live on Payload v4, added schema fuzzing, failure injection and prompt-injection checks, and measured two later versions of the research build.
Status as of 15 November 2026: umbrella issue 19591 is open and its fix pull request 19592 is open. Both were filed on 13 November 2026; the full list is under Upstream status.
What this means for teams running the plugin
Payload 3.93.0 with @payloadcms/plugin-mcp:
- Pass
overrideLock: falseon every update. With it, the tool refused a write to a locked document in 3 of 3 trials, but the refusal is plain text with noisError, so check the answer text. - Keep the delete tool away from agents on collections where an editor’s lock must be respected. It has no
overrideLockargument and deleted a locked document in 10 of 10 trials, and a proxy in front of the server cannot add the argument. An MCP key without the delete capability does not list the delete tool. - Never let an agent send a
whereof{}. Payload treats it as “match everything”: an update changed 17 of 20 documents (our own hook refused the other 3) and a delete removed all 20, in 5 of 5 runs each. - Read the text of every bulk answer. With one document failing, the headline still said “Multiple documents updated” and
isErrorwas unset, in 5 of 5 runs. - Read every answer as text. On 3.x a server-side refusal is ordinary text with
isErrorunset (9 of 9 calls with a missing id), a validation refusal drops the reason REST gives, and a misspelled argument name is dropped without a word. - Collection fields named
locale,draftorselectcannot be written through the 3.x tools: the call answers success and the value is not stored (5 of 5). v4’s rewrite fixes this. - Array edits in localized documents lose the other locales’ row values. This is the known issue 19435; a fix is proposed in pull request 19436.
- No surface, REST included, has a version check, so a write from a stale read silently replaces another editor’s array changes. That is a limit of Payload’s core; an MCP layer can add a best-effort check, as our research build does.
- On v4 main (4.0.0-beta), the lock bypass, the delete of a locked document and the empty
wherebehave the same; we re-ran them live.
Setup and boundary
| Target | @payloadcms/plugin-mcp 3.93.0 (npm latest on 13 November 2026), payloadcms/payload v3.93.0, commit 77425c50, MIT. The 3.x branch had no plugin change since that tag |
| v4 | PAY-01 to PAY-10 re-run live on a separate sandbox built from main ae5e1dc6 (4.0.0-beta), with a raw client and a REST control, no agents. PAY-11 to PAY-13 checked by reading main f4662dd, which changed no package code since ae5e1dc6. The public reproduction also runs on the npm release 4.0.0-beta.9, published later on 13 November |
| Sandbox | Loopback Payload 3.93.0 app, Postgres 17.6, next dev (the Postgres adapter pushes its schema only in development). Two users: editor-a owns the MCP key and is the REST control, always on the same documents; editor-b holds locks and is the second writer. Document locking on, with Payload’s default lock duration of 300 seconds. Our harness created editor-b’s locks as rows in Payload’s locked-documents collection, the row the admin panel writes when a user opens a document, and refreshed them during U8; no browser was used. Deterministic seed |
| Catalog | 12 tools (find, create, update and delete for three collections), no annotations, no server instructions; 4,031 tokens (cl100k tokenizer) |
| Transport | Streamable HTTP. Agent runs and the frozen probes negotiated MCP 2025-11-25; the completeness pass’s own drivers negotiated 2025-03-26. Agents reached the server through our stdio-to-HTTP bridge, which has no version number: a dependency-free file, harness/lib/stdio-bridge.mjs in the experiment repository, that forwards each message unchanged |
| What we changed | A research build: five commits on the plugin (research v1, the benchmarked build), then two more after the benchmark (v2 and v2.1), transpiled into a copy of the installed package |
Agents ran through mcp-audit agent, our agent harness, with built-in tools off and only the MCP tools available: two AI models, model A throughout and model B on set A. Every result was graded by a checker that read the database through REST, never by the agent’s own account.
Everything ran on loopback only, against throwaway sandboxes with a deterministic seed; no hosted service and no one else’s data was touched. Payload has no vendor-hosted MCP service: the plugin runs inside the customer’s own Payload app, so a pinned app and config fix what an agent sees. Token figures come in two kinds: benchmark tokens and cost are provider-reported, and sizes of what the server returns are tokenizer estimates of the text.
Headline finding: update and delete write through document locks (PAY-01, PAY-02)
- PAY-01 severity: High (IsoKron rubric v1.0) · Attribution: MCP-specific · Article label: S2
- PAY-02 severity: High (IsoKron rubric v1.0) · Attribution: MCP-specific · Article label: S2
Payload writes a lock row when a user opens a document in the admin panel and refreshes it while the form is open; by default a lock expires after 300 seconds without a refresh. Its REST endpoints pass overrideLock as false, so another user’s update or delete gets HTTP 423. The plugin’s update tool defaults overrideLock to true, and its delete tool never passes the flag, so the Local API’s default of true applies.
| Arm | 3.93.0 | v4 main ae5e1dc6, live |
|---|---|---|
updatePosts while editor-b holds the lock | Written 10 of 10; editor-b’s lock row deleted 10 of 10; “success” text, no isError | Same: written 10 of 10, lock deleted 10 of 10, isError 0 of 10 |
The same with overrideLock: false | Refused 3 of 3, as plain text without isError | Refused 3 of 3, now with isError |
deletePosts while editor-b holds the lock | Deleted 10 of 10, success text | Deleted 10 of 10; overrideLock refused by the schema |
| Control: REST update and delete as the same user | 423, 10 of 10 each; lock kept | 423, 10 of 10 |
| Research build | Written 0 of 10, deleted 0 of 10; each answer an isError conflict naming editor-b | Not built |
| Our upstream-shaped 3.x patch, scripted replay | Written 0 of 10 with the lock kept (the refusal is still plain text on 3.x); deleted 0 of 10 | On main, the patch’s tests fail 4 of 4 before it and pass 4 of 4 after |
MCP update tool, vendor build
0 refused, 10 went through (10)
MCP delete tool, vendor build
0 refused, 10 went through (10)
REST update (control)
10 refused, 0 went through (10)
REST delete (control)
10 refused, 0 went through (10)
MCP update tool, research build
10 refused, 0 went through (10)
MCP delete tool, research build
10 refused, 0 went through (10)
Source: the headline table above (probes E1 and E2, and their re-runs on the research build).
An unsafe default, not a broken lock. Payload’s documentation says a locked document refuses updates and deletes from other users, and it documents that the Local API, which trusted server code uses, defaults overrideLock to true. The plugin passes that default on, so an agent acting for a signed-in user writes like trusted server code rather than like another user editing through REST. A refuter agent narrowed PAY-01 to exactly that, “an unsafe default with an opt-out”, because the update tool does accept overrideLock: false, and our Discussion asks the maintainers about the default. PAY-02 is stronger: the delete tool has no opt-out. Either way the write also deletes the editor’s lock row.
- What it does to an agent. In the native session (job U8, the opening story), the agent’s stale write went through editor-b’s lock and dropped her saved row, and the agent told the user it was done. On the vendor build U8 then failed 5 of 5 benchmark runs; on the research build it passed 5 of 5, each run refused and telling the user who was editing.
- A proxy cannot fix all of it. A rule in front of an unchanged server that sets
overrideLock: falseon update stopped the write (written 0 of 10, lock kept). The delete cannot be fixed that way: the vendor delete tool has no argument to set, and it still deleted 10 of 10. - Fix. Default
overrideLockto false for the MCP tools only, and let delete take it; the Local API keeps its default. Raised in Discussion 19597, with the pull request prepared.
Second finding: match-all filters and partial bulk failures (PAY-03, PAY-05)
- PAY-03 severity: Critical (IsoKron rubric v1.0) · Attribution: MCP-specific · Article label: S2
- PAY-05 severity: Low (IsoKron rubric v1.0) · Attribution: MCP-specific · Article label: S3
| Input | Through the tool, 3.93.0 | REST control | v4 main, live | Research build |
|---|---|---|---|---|
Update with a where of {} | 17 of 20 documents changed (our archive hook refused 3), 5 of 5 runs; a where of [] did the same | No where: 400 “Missing ‘where’”, 5 of 5 | 17 of 20 changed, 5 of 5 | Refused 5 of 5 unless all: true |
Delete with a where of {} | All 20 deleted, 5 of 5 runs | 400, 5 of 5 | All 20 deleted, 5 of 5 | Refused 5 of 5 |
| Bulk update, one of five documents refused by a hook | Headline “Multiple documents updated”, isError unset, 4 committed: 5 of 5 | 400 with errors; the same 4 committed: 5 of 5 | Fixed: flagged 5 of 5 | PARTIAL headline with isError, 5 of 5 |
| Bulk delete, one of five refused | Success text with “Errors: 1”, no isError (reproduced by hand before filing, 2 of 2) | 400 with errors | Still unflagged: 4 deleted, isError 0 of 5 | PARTIAL headline with isError by design; not probed |
Why {} matters. In Payload’s own Local API, {} is a valid “match everything” filter, so the refuter narrowed this one too: the matching is not wrong; the gap is a missing guard on a destructive match-all write offered to an agent. The REST control is the closest request REST allows, not an identical one: REST cannot express an empty filter, and it refuses a missing one with HTTP 400. The guard is a behavior change, so we will ask in a Discussion before offering a patch.
Why the bulk answer matters. The four committed documents are the same on both surfaces; only the report differs. An agent reads the answer, not the database, and the headline says success. Pull request 19592 sets isError on the bulk delete on main, where the bulk update is already fixed.
MCP layer or platform? The control arm
| What we tested | Through the MCP server | Through Payload’s REST API | Where the problem is |
|---|---|---|---|
| Update a post another user has locked | Written 10 of 10, lock deleted 10 of 10, success text | Refused with HTTP 423, 10 of 10; lock kept | MCP plugin: the update tool defaults overrideLock to true |
| Delete a locked post | Deleted 10 of 10 | Refused with HTTP 423, 10 of 10 | MCP plugin: delete never passes the flag |
Update and delete with a where of {} | 17 of 20 changed; all 20 deleted (5 of 5 runs each) | No where (REST cannot send an empty one): 400 “Missing ‘where’”, 5 of 5 | MCP plugin: the guard only rejects an empty value |
| Bulk update with one failing document | Success headline, no isError, 5 of 5 (4 committed) | 400 with errors, 5 of 5 (the same 4 committed) | MCP plugin: how the result is reported |
Write fields named locale and draft | Not stored, success text, 5 of 5 | Stored, 5 of 5 | MCP plugin, 3.x only |
| Edit an array row in a localized post | Other-locale labels lost 5 of 5; row ids refused by the schema | Kept 5 of 5 (REST sends row ids) | MCP plugin; known issue 19435 |
| Two simultaneous partial updates to different fields of one post | One field lost, 10 of 10 pairs (MCP plus REST) | One field lost, 10 of 10 pairs (REST plus REST) | Platform |
| Write from a stale read | Another editor’s row lost with no signal, 10 of 10 benchmark runs (A4 and U9) | No version check on any surface; on v4 main the stale write lost the row 5 of 5, REST the same | Platform |
The rows are from scripted probes with no model in the loop, each with its REST control, except the stale-write row, which is from the benchmark and the v4 re-run. Where the two surfaces accept different inputs, the control sends the closest REST request and the table says so: REST cannot send an empty where.
What is platform behavior. Two partial updates to different fields of one post, released together, lost one field in 10 of 10 pairs, and two REST updates lost one in 10 of 10 as well. By our house rule that is not an MCP finding, and we report it only so the MCP findings are not overstated. Payload has no version check that a client can send on an update, on any surface.
Payload: the findings
This article reports 13 findings, PAY-01 to PAY-13; the experiment repository lists the same ones as F1 to F13. Each went through seven recorded checks: a reproduction with a raw JSON-RPC client and none of our tooling; a check against upstream main; the code’s history; Payload’s documentation, for intent; a search of prior reports; a refuter agent in a clean context (model A, a two-stage brief, code only); and a classification with a severity. For PAY-01 to PAY-10 those are 69 recorded results on the probes’ hash-chained log. PAY-11 to PAY-13 came from the completeness pass’s schema fuzzing (79 cases over the 9 write tools) and were recorded on that pass’s own log, with one refuter agent.
The v4 column is a live re-run on a sandbox built from main ae5e1dc6 for PAY-01 to PAY-10, each experiment with a REST control. For PAY-11 to PAY-13 it is a reading of the source at main f4662dd, marked “source”, not a live run.
The Severity column is the audit’s original scale: our judgment of operational impact, not a security rating. S2: real harm in normal use. S3: harm in a common case, or a gap with a workaround. S4: lower still; here, a correct refusal that drops its reason. Info: a cost or hygiene item. On that original scale no finding here is rated S1, the highest. The Rubric column rates each finding on IsoKron’s severity rubric v1.0, published after this audit; on the rubric, PAY-03 is Critical: one ordinary call deleted every document in the collection.
| ID | Finding | Measured on 3.93.0 | REST control | v4 main | Refuter | Class | Severity | Rubric | Upstream |
|---|---|---|---|---|---|---|---|---|---|
| PAY-01 | The update tool writes through another user’s lock and deletes the lock row | Written 10 of 10, lock deleted 10 of 10; with overrideLock: false, refused 3 of 3 as plain text without isError | 423, 10 of 10 | Reproduced (10 of 10); the opt-out refusal now sets isError | Narrowed: unsafe default with an opt-out | Unsafe default: the MCP tools inherit the Local API’s default for trusted server code | S2 | High | 19591; Discussion 19597 |
| PAY-02 | The delete tool deletes a locked document and has no overrideLock argument | Deleted 10 of 10 | 423, 10 of 10 | Reproduced (10 of 10) | Confirmed | Bug | S2 | High | 19591; Discussion 19597 |
| PAY-03 | Update and delete with a where of {} match every document | 17 of 20 changed (3 refused by our hook); 20 of 20 deleted | No where: 400, 5 of 5 | Reproduced (5 of 5 each) | Narrowed: {} is match-all in the Local API | Design gap | S2 | Critical | A Discussion, not yet filed |
| PAY-04 | Fields named locale, draft or select (and other names the tools use as arguments) cannot be written | Not stored, success text, 5 of 5; the native job stored the market as null | Stored 5 of 5 | Fixed by the data argument; confirmed 5 of 5 | Rejected: fixed on main | Bug on 3.x; a backport candidate | S3 | High | The 3.x question in 19591; a 3.x patch is ready |
| PAY-05 | Bulk update and delete report success with no isError when some documents fail | 5 of 5 | 400 with errors | Bulk update fixed; bulk delete still unflagged (isError 0 of 5) | Narrowed | Bug | S3 | Low | 19591; pull request 19592 |
| PAY-06 | Array edits lose other locales’ row values | 5 of 5 | Kept 5 of 5 | Row ids now accepted; with ids the other locale was kept 5 of 5 | Rejected: duplicate of 19435 | Cited, not claimed | S3 | High | Comment on 19435 |
| PAY-07 | On 3.x, single-document writes echo the whole stored document | 3,278 echo tokens over 13 native jobs; a 21-byte update answered with 384 bytes | REST can trim with select | Narrowed: only delete by id still echoes (836 bytes); update and create answer ids only | Narrowed | Design gap | Info | Informational | Not filed |
| PAY-08 | No tool annotations on 3.x | 0 of 12 tools annotated | None | Fixed (pull request 17027); 13 of 13 annotated | Rejected: fixed on main | Hygiene on 3.x | Info | Informational | The 3.x question in 19591 |
| PAY-09 | Stale writes are accepted with no signal, on every surface | Native agent lost another editor’s row; no signal | Same: no version check on any surface | Reproduced (row lost 5 of 5, success text) | None | Design gap in Payload’s core; the MCP layer can mitigate it with a best-effort check | S3 | High, platform-inherited | Not filed |
| PAY-10 | Two simultaneous partial updates to different fields lose one | 10 of 10 pairs | 10 of 10 pairs (REST plus REST) | Reproduced (core) | None | Platform behavior, not an MCP finding | Info | High, platform-inherited | Not filed (platform) |
| PAY-11 | Argument names the advertised schema forbids are stripped without a word; a misspelled-only update answers success and changes nothing | 4 of 79 fuzz cases | REST also ignores unknown keys, but advertises no schema | Source: fixed (strict inputs) | Rejected: fixed on main | Bug on 3.x; a backport candidate | S3 | Low | Not filed; follows the 3.x answer |
| PAY-12 | Tool schemas omit the fields’ min, max, maxLength and configured locales, so schema-valid values are refused | 9 of 79 fuzz cases | REST refused 8 of the 9 with a reason; it stored 1 (see below) | Source: still present; main returns the reason and the schema after a refusal | Narrowed: a design gap, 6 tools | Design gap | Info | Low | Not filed |
| PAY-13 | A validation refusal names the field but drops the reason REST gives, with no isError | 12 fuzz cases | REST gives the reason (“9 is greater than the max allowed Value of 5.”) | Source: fixed | Rejected: fixed on main | Bug on 3.x; a backport candidate | S4 | Low | Not filed; follows the 3.x answer |
About the Rubric column: it rates what a failure did to the data. Whose code caused it, and any input the schema or the API would refuse, are noted below and do not lower the level. A loss counts as recoverable only when the audit tested a restore, and exposures are listed with their facts, not rated.
- PAY-02 (High): No restore path found: in the audit’s delete check on the posts collection, which keeps versions, a delete also removed the post’s versions (2 to 0).
- PAY-03 (Critical): One call removed documents it did not name one by one. The original S2 reflects the refuter’s narrowing, that
{}is a valid match-all in Payload’s Local API; the rubric rates what the call did to the data and records whose behavior it is as attribution. MCP-specific: REST, the API Payload offers to clients, cannot send an empty filter and refuses a missing one. No restore path found: the tasks collection this probe used had Payload’s defaults, with no trash and no versions. - PAY-06 (High): Restore path documented, not tested: the posts collection keeps versions, but we did not restore from them. Medium if a restore is confirmed.
- PAY-09 (High, platform-inherited): No surface of Payload has a version check, and REST lost the row the same way.
- PAY-10 (High, platform-inherited): Two REST updates lost a field in 10 of 10 pairs as well. “Not an MCP finding” is the attribution; the level is the impact.
- PAY-11 (Low): Reachability: needs argument names the advertised schema forbids.
- PAY-12 (Low): Low, not Informational: in one of the nine cases the refusal named missing row labels instead of the locale, a false error message.
By class, that is 5 bugs (3 of them on 3.x only, fixed on v4 main), 1 unsafe default, 4 design gaps, 1 hygiene item, 1 known upstream issue cited and 1 platform behavior. In one of PAY-12’s nine cases REST stored the value, and the MCP refusal named missing row labels rather than the locale; in the other eight REST refused the value and gave the reason the MCP answer drops.
Static analysis also produced candidates about the plugin’s dev-only experimental tools, which register only when enabled in the plugin config under NODE_ENV=development. They are outside this article and not filed: v4 removed the source-writing tool families, and fixes for dev-only tools on 3.x are unlikely to be wanted.
How the findings were checked
| Stage | Payload |
|---|---|
| Candidates | Desk analysis of the pinned source (9 candidates on the per-collection find, create, update and delete tools, plus candidates on the dev-only tools outside this article), the native session’s friction register, and later schema fuzzing (79 cases over the 9 write tools) |
| Frozen probes on 3.93.0, each with a REST control | 8 experiments, E1 to E8, with 3 to 10 repetitions per arm, and 5 of them re-run on the research build: 13 runs |
| Completeness probes | 25 completed runs: read-only sweep, database statement trace, schema fuzzing, create-read-update-delete round trips, error grading, failure injection, key scope and prompt injection, and fixed call sequences |
| Re-run on v4 main | PAY-01 to PAY-10 live on ae5e1dc6, with a raw client and a REST control: PAY-01, PAY-02, PAY-03, PAY-09 and PAY-10 reproduced; PAY-05 half fixed; PAY-04 and PAY-08 confirmed fixed; PAY-06 and PAY-07 changed. PAY-11 to PAY-13 checked by reading the source at main f4662dd |
| Recorded checks | All 13 findings: the seven checks above. PAY-01 to PAY-10: 69 recorded results on the probes’ log. PAY-11 to PAY-13: on the completeness pass’s log |
| Refuters | Fresh agents in clean contexts (model A, the model that also ran the agents), code only: PAY-02 confirmed; PAY-01, PAY-03, PAY-05, PAY-07 and PAY-12 narrowed; PAY-04, PAY-06, PAY-08, PAY-11 and PAY-13 rejected as fixed on main or a duplicate; PAY-09 and PAY-10 have no refuter verdict |
| Register | 13 findings: 5 bugs, 1 unsafe default, 4 design gaps, 1 hygiene item, 1 cited upstream issue, 1 platform behavior |
| Hand reproduction | The reproductions run by hand on the audit’s sandbox and, at the same time, on a separate machine; sign-offs recorded on the probe logs |
| Reported upstream | 1 issue, 1 pull request, 1 Discussion and 1 comment on an existing issue, on 13 November |
| Agent series | 144 measured benchmark runs; 52 control runs (auditing on with per-run anchors, forced compaction, prompt injection) and 8 fault runs; 41 agent records comparing research v2 with v1 (stopped early by a billing refusal); 60 runs comparing research v2.1 with v1 |
- FindDesk analysis, the native session and schema fuzzing on the pinned 3.93.0 build
- Probe with a control8 frozen experiments and 25 completeness runs, each against the same operation over REST
- Check and refuteRaw-client reproduction, v4 checks and refuter agents in clean contexts; 13 findings registered, including those narrowed, fixed on main or already known upstream
- Reproduce by handOn the audit’s sandbox and on a separate machine, with sign-offs recorded
- File1 issue, 1 pull request, 1 Discussion and 1 comment on 13 November
Source: the stage table above.
What the checking caught, in our own records. The recorded classification text for PAY-01 says 9 of 9: it was drafted from a dry run before the frozen run finished, and the frozen run recorded 10 of 10. The log keeps the text as written; the audit report records the correction. Checked against the per-run table before this article, three hand-written figures were also corrected: the false “done” count, and the model B and observer totals, which had included A3. The figures below are the corrected ones. The v4 re-run also showed that one main-branch code reading was wrong: on main, update and create no longer echo the document (PAY-07 narrowed to delete).
What held up
- Access rules. The plugin passes the key’s user with
overrideAccess: false, so collection access rules apply. A key scoped to three tools listed exactly those three; calls to other tools were refused with nothing changed (4 of 4); a capability switched off mid-session was refused on the next call; a disabled, wrong or missing key got 401. The one place MCP let a user do more than REST is the document lock. - Partial updates. A partial update never cleared the fields it did not name, on either build or over REST.
- Transactions. A server killed mid-write rolled back: 0 of 3 writes landed, on both builds and in the controls.
- Reads. Every find answer’s
Totalmatched REST, and no find answer contained an e-mail address. A list read costs 4 database statements against REST’s 3, with no fan-out and no repeated statements. - Retries before a write. After an injected 504 in the native session, and after 429, 401 and 503 errors in the fault runs (6 of 6), the agent re-read state before retrying and never duplicated.
- Agent reports. Most vendor-build failures in the benchmark were honest or unclear, not false successes (the false “done” labels are discussed under the benchmark).
Usability pass: what the server costs an agent
Before designing any change, model A used the vendor build for 13 everyday jobs, one run each on a fresh seed, with only the 12 MCP tools: no shell, no files, and no mention of an audit. A Phase 1 baseline run came first.
| Session | Jobs passed | Job tool calls | Response tokens | Cost |
|---|---|---|---|---|
| 13 everyday jobs, one run each | 9 of 13 | 44 | 11,434 | 0.62 USD (14 runs, the baseline run included) |
It failed 4: U2 (the localized row labels were lost, issue 19435), U6 (the market field locale was stored as null; the agent noticed and told the user), U8 (the opening story: a stale write through the lock, success claimed) and U9 (a stale array write dropped another editor’s row). After an injected 504 in U7, the agent checked before retrying and did not duplicate. U3 and U13 passed because the agent found a refused document in plain error text that carried no isError.
Of the 11,434 response tokens, write echoes were 3,278; the other 8,156 were find results and short answers, which our toolkit counts as fully needed because it cannot split Payload’s prose-plus-JSON answers, so this split is coarse. Each session start cost 4,195 tokens before the first call, 4,149 of them for the tool list.
13 everyday jobs, 44 tool calls
3278 write echo, 8156 everything else (11434)
Source: the session’s response tokens, as split in the paragraph above.
The friction register has 9 items: the lock takeover, the stale write, the localized rows (the known issue), the colliding field names, write echoes, catalog weight, missing isError, no instructions or annotations, and writes one document at a time. The change list has 7 items, all built into the research build; catalog weight was measured and held: the research build’s new arguments add about a thousand tokens, and removing the schema each tool repeats would need shared schema definitions that the MCP SDK the plugin uses does not support.
The Payload research build
Five commits on the pinned plugin, containing every change from the session plus the bug fixes:
- Locks respected by default, as on REST:
overrideLockdefaults to false, and delete accepts it. A locked write returns anisErrorconflict that names the holder and gives the lock’s age, the document’supdatedAtand the current document, with the instruction not to take over unless the user asked. - A best-effort version check: an optional
ifUpdatedAt, plus the version this user last read as an implicit check. A stale write gets a conflict carrying the current document. This is read, then write, in two steps, not an atomic compare-and-set; that needs support in Payload’s core. - Array rows keep an optional
id, equivalent to pull request 19436. - A
dataargument for fields whose names collide with tool arguments (additive; nothing renamed). - Compact write receipts (
id,updatedAt, status, fields written);returnDocument: truegives the old full answer. isErroron every error, and bulk answers with one result per document under a PARTIAL headline.- Six lines of default server instructions and tool annotations.
- A
whereof{}is refused unlessall: trueis passed.
The research catalog is 5,060 cl100k tokens against 4,031, 25% more. Re-run on the research build, the lock probes wrote 0 of 10 and deleted 0 of 10, each answer an isError conflict naming editor-b (10 of 10); the empty-filter update and delete were refused 5 of 5; the partial bulk update answered PARTIAL with isError 5 of 5.
After the benchmark the build was also type-checked and tested inside Payload’s monorepo: 0 type errors in a strict build. Payload’s own plugin integration suite passes 46 of 74 as is; all 28 failures assert the old full-document write text that the compact receipts replace by design, and with returnDocument: true in those calls it passes 74 of 74.
Benchmark: vendor build against research build
We ran 144 measured runs on 13 November, from 07:29 to 08:16 UTC, in a seeded shuffled order alternating builds, each from a fresh seed. Model A ran 5 runs per task and build on set A (60 runs, A3 included) and set B (50). Model B ran 2 per task and build on set A (24, A3 included). An observer arm with our capture proxy off ran 1 per task and build on set A (10, A3 included, A4 excluded). The tables regroup them: set A’s five pre-registered tasks (58 runs, model A and the observer), set B with A3 (62 runs), and model B’s 20 runs on the pre-registered tasks, in the text; model B’s 4 A3 runs passed on both builds and are in no table. The benchmark cost 5.44 USD in model spend. Correctness is a checker reading the database through REST after each run, never the agent’s account.
First, the limits. Probe counts such as 10 of 10 show that a behavior occurs under the stated conditions, not how often. The benchmark runs each task 5 or 6 times per build, which shows spread, not a rate, and the two builds were measured as wholes, with no fixes-only arm. Completion intervals are 95% Wilson score intervals with runs pooled; intervals for differences in tokens and cost come from a task-clustered bootstrap (10,000 resamples of tasks, seed 1). The full list is under Benchmark limits.
Set A: tasks fixed before any change was designed
Set A’s prompts and checkers were committed at 06:18 UTC on 13 November, before any research-build code was written, and fixed by the pre-registration freeze at 06:43 UTC, on the evidence chain of the native-session series. A4’s seed and description changed with our toolkit’s conflict rule before that freeze and before any agent run. The benchmark series itself was frozen at 07:28 UTC, after the change list was written; the measured runs followed. The fair test is five tasks: A1, A2, A4, A5 and A6 (A3 is explained below).
| Task | Vendor build | Research build | What decided it |
|---|---|---|---|
| A1 Publish a post and add a task | 5 of 6 (one credit error) | 6 of 6 | None |
| A2 Change one array row’s link | 0 of 6 | 6 of 6 | Other-locale labels lost on the vendor build (issue 19435); kept with row ids |
| A4 Add a row; editor-b adds hers after the agent’s read | 0 of 5 | 5 of 5 | Vendor: the stale write drops her row. Research: a conflict, then an informed retry |
| A5 Delete one person’s done tasks | 6 of 6 | 5 of 6 (one credit error) | None |
| A6 Add a listing for the de-DE market | 0 of 6 | 6 of 6 | Vendor: the locale field cannot be written. Research: data |
| Total | 11 of 29 (38%, 95% interval 23% to 56%) | 28 of 29 (97%, 83% to 99%) | None |
| Tokens over all attempts | 984,642 | 1,159,007 | None |
| Expected tokens per verified success | 89,513 | 41,393 | None |
Two runs, one per build, failed with an API credit error on our account. Both totals count them as failed attempts with zero tokens; they were not graded.
Each row is drawn to its own scale.
Source: the set A table above.
Per task, model A and the observer arm (medians from our toolkit’s report, which counts the two credit-error runs as zero-token attempts; leaving them out does not change the token and cost conclusions below):
| Task | Tool calls, vendor → research | Tokens per run, vendor → research | Cost per run, USD, vendor → research |
|---|---|---|---|
| A1 Publish a post and add a task | 4 → 4 | 35,385 → 41,240 | 0.0441 → 0.0439 |
| A2 Change one array row’s link | 2 → 2 | 34,373 → 40,106 | 0.0337 → 0.0344 |
| A4 Add a row after another editor’s change | 2 → 3 | 34,294 → 55,816 | 0.0329 → 0.0504 |
| A5 Delete one person’s done tasks | 3 → 3 | 36,980 → 40,757 | 0.0374 → 0.0366 |
| A6 Add a listing for one market | 2 → 1 | 33,659 → 26,342 | 0.0338 → 0.0281 |
The research build costs more where it adds work (A4’s conflict and retry) and less where data lets one call do the job (A6); the catalog it adds is re-read from the prompt cache on every turn.
“Expected tokens per verified success” is the tokens of every completed attempt divided by the runs a checker verified: 984,642 over 11 on the vendor build and 1,159,007 over 28 on the research build. It roughly halved because far more runs succeeded; tokens per run went up. The median run used 34,373 tokens on the vendor build and 40,709 on the research build, a difference of 6,336 with an interval of −7,445 to 21,477. Mean cost per run moved from 0.0351 to 0.0376 USD; the interval for that difference, −0.0037 to 0.0102, includes zero. The audit attributes the rise to the larger catalog and instructions and to the extra calls of conflict recovery; that split was not measured separately.
With five tasks, a per-task sign test cannot show much: 4 tasks improved and 1 got worse, p = 0.375, not significant. The one that got worse, A5, lost a run to the API credit error, as A1 did on the vendor build. At run level, McNemar’s exact test pairs each task’s vendor and research runs by run order: 18 pairs favored the research build, 1 the vendor build and 10 were tied, p = 0.0000763. It treats the 29 pairs as independent although they cluster in five tasks, so it describes these runs and does not show that the result carries over to other CMS work.
Model A alone on the five pre-registered tasks, without the observer arm, went from 9 of 25 to 24 of 25, with medians of 34,365 and 40,709 tokens and the same interval for the difference. Model B, 2 runs per task, went from 4 of 10 to 10 of 10 on the five pre-registered tasks, failing the same three on the vendor build (A2, A4, A6). The observer arm, with our capture proxy off and one run per task (A4 had none), showed the same pattern (2 of 4 against 4 of 4), and a later arm with auditing on and an anchor after every run gave it again (4 of 10 against 10 of 10, 2 runs per task), so the capture did not change the outcomes.
No run on either build chose a wrong tool (0 of 158 runs: the 144 benchmark runs and the 14 native-session runs). What failed on the vendor build was the first write’s arguments: for model A on the pre-registered tasks, the first write was valid on the first try in 21 of 28 runs against 28 of 28, mostly the field-name trap of A6 and row ids the vendor schema refuses. The vendor build’s failures were also silent: 17 model A runs and 6 model B runs on these tasks failed their checker with no refusal returned, against none on the research build.
Of the 17 graded vendor-build failures on the five pre-registered tasks, 8 were honest (the agent said it could not do the job), 7 were unclear and 2 were classed as false “done” claims, both on A6, where the market field could not be written. The label is mechanical. The job asked for “a finished offer”, and both answers called the listing “finished”, which our claim rule reads as a done claim. Both then told the user the saved record showed no market, so the agent did not hide the failure.
A3: a pre-registration defect, disclosed
At the pre-registration freeze (06:43 UTC), A3’s prompt file was a copy of a different job’s prompt (“Every task that isn’t done yet should be priority 3 from now on…”), while A3’s frozen description and its checker were “mark every ‘doing’ task as done; say if any could not be changed”. The prompt did not match its own pass criterion. A dry run of the benchmark, not measured, caught it: the agent set priorities, and the checker failed it. After the pre-registration freeze, but before the benchmark freeze (07:28 UTC) and so before any measured run, we changed the prompt to match the frozen description: “Please mark every task that is currently ‘doing’ as done, and tell me if any of them couldn’t be changed.”
A task changed after its pre-registration freeze cannot count as pre-registered, and our toolkit refused it as one (“prompt changed since the freeze”). So A3 is reported with the friction-derived set below. It passed 6 of 6 on both builds and does not touch the set A totals above.
Set B: tasks from the session (favors the research build by construction)
Set B is the session jobs that failed, plus a clean-up job and A3. These tasks were chosen after the changes were designed, from the session that produced them, so they favor the research build by construction.
| Task | Vendor build | Research build |
|---|---|---|
| U2 Localized row edit | 0 of 5 | 5 of 5 |
| U6 Three listings, market en-GB | 0 of 5 | 5 of 5 |
| U8 Add a row while editor-b keeps the post open | 0 of 5 (lock taken over, her row lost) | 5 of 5 (lock respected, honest answer) |
| U9 Add a row after a stale read | 0 of 5 | 5 of 5 |
| U11 Scratch clean-up | 5 of 5 | 5 of 5 (more calls: one delete per document; see the appendix) |
| A3 (moved here) | 6 of 6 | 6 of 6 |
| Total | 11 of 31 (35%, 95% interval 21% to 53%) | 31 of 31 (100%, 89% to 100%) |
| Tokens over all attempts | 1,203,009 | 1,619,130 |
| Expected tokens per verified success | 109,364 | 52,230 |
Write echoes shrank where writes dominate (A6: 419 to 341 response bytes per run; U2: 1,486 to 995) and grew where conflict answers carry the current document (U8: 1,563 to 5,360).
Conflict recovery
Three tasks seed a conflict right after the agent’s first read: in A4 and U9 another editor adds a row and leaves; in U8 editor-b saves a row and keeps the post open, holding the lock to the end of the run. Our toolkit classifies each run from the agent’s own calls under its conflict rules. The recovery rate is runs that re-read (or used the current state the conflict answer carried), kept the other change and landed their own, out of the conflicts the build signaled. We report version 3 of those rules, re-applied on 13 November to the saved records of every run; the version 2 records made during the benchmark sit beside them.
The research build recovered 10 of the 15 seeded conflicts: A4 5 of 5 and U9 5 of 5, with a median of 1 extra call and about 60 extra response tokens per recovery. Model B recovered A4 2 of 2. The vendor build never returned a conflict: no signal in 15 of 15 seeded runs (2 of 2 for model B), and the other writer’s change was lost every time.
U8, the other 5. In all five research runs editor-b held her lock to the end; the agent’s writes were refused with a conflict naming her, it did not take over, and it told the user who was editing. Our checker passed all five. They are recorded as “gave up and reported the failure” (gave_up_reported_failure), which counts against the rate. The rules have a separate class, “conflict persisted”, for a run that ends with the other writer still holding the lock, and it sits outside the rate; our U8 checker was frozen before that class existed and could not report whose lock remained, so 0 runs were recorded in it. A later series with an updated checker recorded five new U8 runs on the same build as “conflict persisted”, 5 of 5. The appendix gives the detail; the benchmark’s recorded classes stand.
Vendor build, 15 seeded
0 recovered, 0 refused, reported who was editing, 15 no signal, other change lost (15)
Research build, 15 seeded
10 recovered, 5 refused, reported who was editing, 0 no signal, other change lost (15)
Source: version 3 of our toolkit’s conflict rules, re-applied to the saved records, in the text.
Benchmark limits
- Five or six runs per task and build (two for model B); repetitions measure spread within one prompt, not variety across CMS work. Statistics with five tasks cannot show much, and the run-level test treats run pairs as independent.
- Set A’s prompts were committed before any change was designed, but A4’s seed changed before the pre-registration freeze, and A3’s prompt after it, before any measured run (A3 moved to set B). Set B favors the research build by construction.
- The research build is measured as a whole, with no fixes-only arm. We did not split the change by type: no ablation and no accounting model was run, so which change produced which gain is not measured, beyond the per-task reasons in the table.
- Prompt caching carries across runs; the shuffled order exposes both builds equally. Two runs were lost to a billing error and count as failures.
- One client, two models, one sandbox, agents running as one admin user. Other clients and pricing will give other numbers.
After the benchmark. The completeness pass found that research v1 made batching worse (U11 in one call in 0 of 10 opportunities, against 8 of 10 on the vendor build) and that its bulk calls had no version check. Research v2 fixed both but lost editor-b’s change in 6 of 9 conflict runs, and research v2.1 fixed that. In a final series, research v2.1 batched U11 in 10 of 10 opportunities and kept editor-b’s change in 15 of 15 conflict runs, as research v1 did. The benchmark figures above are for research v1. The appendix has the history.
Other checks from the completeness pass
- Retries after a failure. A retry is safe when the failure came before the write. After a write had landed, a blind retry of a create made a second copy (5 of 5 after a client timeout) unless a unique field refused it, and a delete delivered twice answered “Not Found” for a delete that had succeeded (5 of 5). Neither surface has an idempotency key. A real agent re-read state before retrying after 429, 401 and 503 errors and never duplicated (6 of 6), but could not see a duplicate the network made (0 of 2).
- Prompt injection. Stored text addressed to the assistant came back verbatim and unmarked inside the find results on both builds. The agent acted on it in 0 of 12 runs, but never warned the user, and in 9 of 12 it left the passage out of its answer without saying so.
- Fixed call sequences. Without a model, the best fixed sequence of calls passes 3 of 6 set A tasks on the vendor build and 6 of 6 on the research build.
- Compaction. A forced mid-task compaction changed no pass pattern on the research build.
- Spend. Model spend charged for the whole audit was about 11.58 USD; the last two series ran on a subscription plan, about 4.77 USD of estimated usage.
Upstream status
The linked items below were filed on 13 November 2026, after we reproduced the filed cases by hand and read every patch. A minimal public reproduction of PAY-01, PAY-02 and PAY-05, on 3.93.0 and on 4.0.0-beta.9, is in isokron-ai/payload-mcp-repro.
| Findings | Upstream item | Status as of 15 November 2026 |
|---|---|---|
| PAY-01, PAY-02, PAY-05 | Umbrella issue 19591: update and delete write through document locks, and partial bulk failures report success; asks whether the maintainers want 3.x fixes | Open; labeled Bug and plugin: mcp by Payload’s triage automation, no reply yet |
| PAY-05 | Pull request 19592: isError when a bulk delete fails or partly fails, on main, with tests that fail before and pass after | Open, ready for review with its checks passing, awaiting maintainer review |
| PAY-01, PAY-02 | Discussion 19597: should the MCP update and delete tools respect document locks by default, like REST? A pull request with tests is prepared and waits for a reply there, or a week | Opened 13 November; no comments yet |
| PAY-06 | A measured reproduction on the known issue 19435, offering to test its fix | Posted 13 November |
| PAY-03 | A Discussion on a guard for a match-all where in update and delete; a pull request only if the maintainers agree | Not yet filed; next in our filing order |
| PAY-04, PAY-08, PAY-11, PAY-13 | Backports from v4 to 3.x, only if the maintainers want 3.x fixes. Issue 19591 asks, naming PAY-04 and PAY-08; a 3.x patch for PAY-04 is ready, and PAY-11 and PAY-13 would follow the same answer | Waiting on that answer |
| PAY-07, PAY-09, PAY-10, PAY-12 | Not filed: two design gaps, a possible version-check proposal for v4, and platform behavior | None |
As of 14 November 2026, no maintainer had replied to these filed items; the labels came from Payload’s triage automation. We file at most two new threads a day and keep at most two of our pull requests open and unreviewed. We will record maintainer responses and fixes in the changelog below. Maintainers are not obligated to review, accept or merge our submissions.
Reproduce it and rebuild the builds
Reproduce the Payload findings. isokron-ai/payload-mcp-repro is Payload’s blank template with one posts collection, two admin users and the MCP plugin, on SQLite. After pnpm i && pnpm seed && pnpm dev, node scripts/repro-locks.mjs, scripts/repro-bulk-delete.mjs and scripts/repro-bulk-update.mjs run PAY-01, PAY-02 and PAY-05, each with its REST control; v4-beta/ is the same app on 4.0.0-beta.9. The full harness, with the Postgres sandbox, the probes, the tasks, the checkers and the stdio bridge, is in the experiment repository below.
Rebuild the builds. The patch series, the three research builds (v1, v2 and v2.1), the harness, the findings register and the benchmark reports are in isokron-ai/payload-mcp-writepath-experiment, with Payload’s MIT license and upstream’s plugin source at v3.93.0. Its harness/build/build.sh checks the upstream import and the patch series, and rebuilds the 7 changed files of each build byte-identical to the builds we measured. The research builds are experiments, not upstream releases: use Payload’s own plugin for real work.
Evidence
Each Payload series was frozen before its runs (the harness, seed, checkers, prompts and toolkit commit hashed), recorded on a hash-chained log, and anchored off-box at its freeze, most of them also with RFC 3161 timestamps from two independent authorities. The hand sign-offs are recorded on the two probe logs. There are nine series. Seven are closed into Evidence Packs, and each verifies with our toolkit’s verify command: the native session (with the baseline run), the frozen probes, the benchmark, the completeness probes, the completeness agent arms, the fault re-runs and the research v2.1 agent series. The other two, the research v2 batching series and the v2.1 probes, are recorded and anchored but not yet closed. The packs are not published with this article, so readers cannot verify them today; How to verify an MCP audit report yourself shows what a pack contains and how one is checked.
What this does not show
- Small counts. 3 to 12 trials or pairs per scripted arm, and 5 runs per benchmark task and build (2 for model B). Deterministic code paths (10 of 10, 5 of 5) show a mechanism under the stated conditions, not how often it happens; only the simultaneous pairs are races.
- No natural collisions. Every agent run was one agent alone, with the other writer seeded by our harness at a chosen moment. editor-b’s admin-panel save was simulated as a REST update of the items she saw when she opened the post, and her lock was written by our harness; the admin panel’s own handling of a stolen lock was not exercised.
- Combined builds and favorable tasks. Each build was measured as a whole, with no fixes-only arm, and set B favors the research build by construction. The research build’s version check is read-then-write, not atomic. The benchmarked build is the transpiled research v1; v2.1 was measured only on six tasks in one later series. Both builds protect only writes made through them: a REST or admin-panel writer can still overwrite an agent’s change.
- Refuters on the same model. The refuter agents ran on model A, the model that also ran the agents. Clean contexts keep a refuter from seeing our reasoning, but the same model can share the same blind spots; our GitHub audit used refuters on a different model.
- One sandbox. Payload ran under
next dev(both builds), on one Postgres; v4 was re-run live with a raw client only, with no agent runs and no research build, and PAY-11 to PAY-13 were checked on v4 by reading the source only. Payload keeps no request log, so its target-side evidence is our REST snapshots. One client, two models; other MCP clients may present the tools differently. - Evidence. Seven of the nine Payload evidence series are closed and verify, but the packs are not published, and two series are not yet closed (Evidence).
About this research
IsoKron performs fixed-price MCP server audits with this method: probes against a pinned build with REST controls, a native agent usability pass, clean-context refutation, re-checks against upstream main, tested patches and filing upstream; the before/after benchmark is an optional, quoted add-on. Details: MCP server audits. We also build MCP servers and plugins to the rules in Send the change, not the object; we do not present a test of code we built as independent.
Appendix: the research builds after the benchmark
The completeness pass found one thing research v1, the benchmarked build, made worse. On U11 (delete three scratch tasks and three temporary listings), model A handled the batch in one call in 8 of 10 opportunities on the vendor build and 0 of 10 on research v1, because research v1’s own instruction told agents to pass ifUpdatedAt, one document’s version, on every update and delete. Reading research v1’s code for the fix found a worse gap: its read-version check ran only on writes by id, so a bulk call by where had no precondition. Probes showed research v1’s bulk calls deleting or overwriting a document changed after the read in every valid repetition (5 of 5 for delete, 4 of 4 for update).
Research v2 changed the instruction (several documents in one call with where; every update and delete, bulk included, checked per document against the version last read) and added the check to bulk calls. Batching came back (U11 8 of 8 opportunities, A5 5 of 5), but research v2 lost editor-b’s change in 6 of 9 conflict-guard runs. Each of the six first read the post with a select that left out createdAt, which the memory of the version last read is keyed on, so the write had no precondition. Research v1 had the same gap, hidden because its agents passed ifUpdatedAt explicitly. Research v2.1 keeps that memory through a narrow select: in a probe of that read, the stale write landed 5 of 5 times on research v2 and was refused 5 of 5 on research v2.1.
In a final series that night (model A, 5 runs per task and build, 60 of 60 completed), research v2.1 batched U11 in 10 of 10 opportunities and A5 in 5 of 5 (research v1: 0 of 10 and 0 of 5). It kept editor-b’s change in 15 of 15 conflict-guard runs on A4, U9 and U8, and research v1 did too in this series (15 of 15). Every research v2.1 first write was refused by the server’s read-version check with no ifUpdatedAt from the agent, including 7 runs that first read with the narrow select behind research v2’s losses. Batching cut calls (U11 from 8 to 4, A5 from 3 to 2) but not tokens per run, because research v2.1’s larger session start is re-read on every turn. These are small cells with one model, reported as counts. The benchmark figures in the body are for research v1; research v2.1 was not run on the full task sets.
Appendix: how the U8 runs were classified
Version 3 of our conflict rules counts a run as “conflict persisted”, outside the rate, only when the task’s checker reports that the other writer’s own lock was still in place at the end. Our U8 checker was frozen before version 3 existed and recorded only how many lock rows remained (one in each research run), not whose. We tried to fill the field afterwards from the saved end state, but editor-b’s identity is not in it, so the field stays empty and the five runs count as gave_up_reported_failure, inside the rate.
The saved state does suggest what happened: in every research U8 run exactly one lock row remained, the only lock rows created in the runs came from editor-b’s setup, and the server’s last conflict answer, seconds before the end, named her as the holder. That makes it very likely these were persisted conflicts handled correctly, but that is an inference, not a measurement, and we do not report the benchmark’s rate as 10 of 10 with 5 persisted. The U8 checker was changed after the benchmark freeze to record lock owners. In a later series the same night, with that checker, five new U8 runs on the benchmarked build were recorded as conflict_persisted in 5 of 5: refused three times while editor-b held her lock, nothing forced, the user told the post was being edited. That supports the inference; it does not change the benchmark’s recorded classes.
Changelog
- 14 November 2026: published.
- 14 November 2026: added IsoKron rubric v1.0 ratings beside the original labels: rating lines under the headline and second findings, a Rubric column in the findings table, and notes on the ratings after it. The sentence that no finding is rated S1 now says it refers to the original scale; on the rubric PAY-03 is Critical.