Skip to content
IsoKron

Research / Public audit · Write correctness

Payload MCP plugin 3.93.0: updates and deletes write through another editor’s document lock

Payload’s official MCP plugin inherits a Local API default for trusted server code, so its update and delete tools write through document locks that Payload’s REST API enforces. We measured every finding against the REST API, re-ran them on Payload v4, built and benchmarked a research build of the plugin, and filed upstream.

Test setup
Protocol
MCP 2025-11-25 over Streamable HTTP (`POST /api/mcp`, stateless) in the agent runs and the frozen probes; the completeness pass's own scripted drivers negotiated 2025-03-26. Agents reached the server through our stdio-to-HTTP bridge, an unversioned harness file published in the experiment repository that forwards messages unchanged
Software
@payloadcms/plugin-mcp 3.93.0 (payloadcms/payload v3.93.0, commit 77425c50; npm `latest` on 13 November 2026) on a loopback Payload 3.93.0 app with Postgres 17.6 under next dev; v4 main ae5e1dc6 (4.0.0-beta) re-run live with a raw client, no agents, for PAY-01 to PAY-10, and main f4662dd (no package code changed since ae5e1dc6) read for PAY-11 to PAY-13; the public reproduction also runs on npm 4.0.0-beta.9
Client
Scripted probe clients and REST control arms, no model in the loop; native usability session and benchmark: AI agents through our mcp-audit agent harness, model A throughout plus a second model, model B, on set A; refuter agents on model A

Key results

What we found
In @payloadcms/plugin-mcp 3.93.0, the update tool wrote through another user’s document lock and deleted the lock in 10 of 10 trials, and the delete tool deleted a locked document in 10 of 10, where Payload’s REST API refused every one with HTTP 423; both still did on v4 main. The tools inherit the Local API’s default for trusted server code. A where of {} matched every document: an update changed 17 of 20 (our archive hook refused 3) and a delete removed all 20. Partial bulk failures reported success. Two lost-write behaviors are Payload’s platform, not the plugin: no surface has a version check, and two simultaneous partial updates to different fields lost one in 10 of 10 pairs through REST alone.
Upstream, as of 15 November 2026
Filed on 13 November 2026: umbrella issue 19591 is open, fix pull request 19592 for the bulk delete is open, awaiting maintainer review, Discussion 19597 asks about the lock default, and a measured reproduction is on the known issue 19435. As of 14 November 2026, no maintainer had replied to our issue, pull request or Discussion; Payload’s triage automation had labeled the issue.
If you run this server
On Payload 3.93.0, pass overrideLock: false on every update and read the answer text, keep the delete tool away from agents where editors’ locks matter, never send a where of {}, and read every bulk answer for failures. What this means for teams.
Terms used here
Lost update
Two writers change the same object at nearly the same moment. Both are told the write succeeded, but one change is gone afterwards.
Control
The same change sent through the vendor’s own API, without the MCP server. If the control loses nothing, the MCP server causes the loss. If the control loses writes too, the platform behind it has the problem as well, and the MCP server can still add to it, for example by switching off a check the API offers.
Refuter
A separate AI agent, started in a clean context without the conversation that produced a finding, that tries to break it using the claim, the source code and the reproduction or its captures. It confirms the finding, narrows it to the conditions where it holds, or refutes it.
Version check
A save carries the version it was based on, and the server rejects it if someone saved in between, instead of overwriting their change.
Read-merge-write
Read the whole object, change one part of it, write the whole object back. Without a version check, a change someone else made in between is overwritten.
Pre-registered tasks
Tasks and checks written and frozen before the changes were designed, so they were not adjusted to the changes or to the results. Frozen tasks can still exercise behaviors the changes target; each article says how its tasks were chosen.
End-state check
A run is graded by reading what actually changed in the product afterwards, not by what the agent said it did.
Tool annotation
A hint a tool declares about itself, such as destructiveHint. Clients use annotations for confirmation prompts; nothing enforces them.
cl100k tokens
Token counts from the cl100k tokenizer, used as a common baseline. It is not the tested models’ own tokenizer, so the figures are estimates of size.
Hash-chained log
A log in which each entry includes the hash of the one before, so a later edit to any entry shows.
JSON-RPC
The message format MCP clients and servers exchange. Capturing it raw means every number comes from what was actually sent, not from the agent’s account.
RFC 3161 timestamp
A signed receipt from an independent timestamp authority showing that a given hash existed no later than a given time.

Self-funded research by IsoKron, conducted and written with AI assistance. Not commissioned or endorsed by Payload; IsoKron is not affiliated with Payload. IsoKron sells MCP server audits and builds.

An editor, editor-b, has a post open in Payload’s admin panel, so Payload holds a document lock for her. An agent is asked to add a row, Loppers, to the post’s list, and reads the post. Just after that read, editor-b saves a row of her own, Wheelbarrow, and keeps editing. The agent then writes back the list it read, plus Loppers, through updatePosts, the update tool of Payload’s official MCP plugin, @payloadcms/plugin-mcp 3.93.0. Payload’s REST API refuses such a write with HTTP 423 while another user holds the lock. The MCP tool let it through, deleted editor-b’s lock row and replaced the list, so Wheelbarrow was gone, and the agent told the user it was done.

Two things combined there. The row was lost because the agent wrote from a read taken before her save, and no Payload surface checks for that. The lock is what should have stopped the write: REST enforces it, while the plugin inherits the default of Payload’s Local API, which lets trusted server code write through locks. Her admin-panel session was simulated: our harness held her lock the way the admin panel does and made her save through REST, so what her open form would do next was not tested. That was one native session run; on the vendor build the same task then failed 5 of 5 benchmark runs.

The control arm, the same change sent straight to Payload’s REST API, tells the plugin’s defects apart from the platform’s. If REST loses nothing, the MCP layer causes the loss; if REST loses the change too, the platform has the problem, and the MCP layer can still add to it. The lock bypass is in the plugin: REST enforces the lock. A stale write, and two simultaneous partial updates to different fields, lose changes through REST as well, so those are Payload’s platform behavior; we report them so the plugin’s findings are not overstated.

We audited Payload’s plugin on 13 November 2026: scripted probes against a pinned build with REST controls, refuter agents, a native agent session, a research build with every change, and a 144-run before/after benchmark. A completeness pass the same evening re-ran the first ten findings live on Payload v4, added schema fuzzing, failure injection and prompt-injection checks, and measured two later versions of the research build.

Status as of 15 November 2026: umbrella issue 19591 is open and its fix pull request 19592 is open. Both were filed on 13 November 2026; the full list is under Upstream status.

What this means for teams running the plugin⁠

Payload 3.93.0 with @payloadcms/plugin-mcp:

  • Pass overrideLock: false on every update. With it, the tool refused a write to a locked document in 3 of 3 trials, but the refusal is plain text with no isError, so check the answer text.
  • Keep the delete tool away from agents on collections where an editor’s lock must be respected. It has no overrideLock argument and deleted a locked document in 10 of 10 trials, and a proxy in front of the server cannot add the argument. An MCP key without the delete capability does not list the delete tool.
  • Never let an agent send a where of {}. Payload treats it as “match everything”: an update changed 17 of 20 documents (our own hook refused the other 3) and a delete removed all 20, in 5 of 5 runs each.
  • Read the text of every bulk answer. With one document failing, the headline still said “Multiple documents updated” and isError was unset, in 5 of 5 runs.
  • Read every answer as text. On 3.x a server-side refusal is ordinary text with isError unset (9 of 9 calls with a missing id), a validation refusal drops the reason REST gives, and a misspelled argument name is dropped without a word.
  • Collection fields named locale, draft or select cannot be written through the 3.x tools: the call answers success and the value is not stored (5 of 5). v4’s rewrite fixes this.
  • Array edits in localized documents lose the other locales’ row values. This is the known issue 19435; a fix is proposed in pull request 19436.
  • No surface, REST included, has a version check, so a write from a stale read silently replaces another editor’s array changes. That is a limit of Payload’s core; an MCP layer can add a best-effort check, as our research build does.
  • On v4 main (4.0.0-beta), the lock bypass, the delete of a locked document and the empty where behave the same; we re-ran them live.

Setup and boundary⁠

Target@payloadcms/plugin-mcp 3.93.0 (npm latest on 13 November 2026), payloadcms/payload v3.93.0, commit 77425c50, MIT. The 3.x branch had no plugin change since that tag
v4PAY-01 to PAY-10 re-run live on a separate sandbox built from main ae5e1dc6 (4.0.0-beta), with a raw client and a REST control, no agents. PAY-11 to PAY-13 checked by reading main f4662dd, which changed no package code since ae5e1dc6. The public reproduction also runs on the npm release 4.0.0-beta.9, published later on 13 November
SandboxLoopback Payload 3.93.0 app, Postgres 17.6, next dev (the Postgres adapter pushes its schema only in development). Two users: editor-a owns the MCP key and is the REST control, always on the same documents; editor-b holds locks and is the second writer. Document locking on, with Payload’s default lock duration of 300 seconds. Our harness created editor-b’s locks as rows in Payload’s locked-documents collection, the row the admin panel writes when a user opens a document, and refreshed them during U8; no browser was used. Deterministic seed
Catalog12 tools (find, create, update and delete for three collections), no annotations, no server instructions; 4,031 tokens (cl100k tokenizer)
TransportStreamable HTTP. Agent runs and the frozen probes negotiated MCP 2025-11-25; the completeness pass’s own drivers negotiated 2025-03-26. Agents reached the server through our stdio-to-HTTP bridge, which has no version number: a dependency-free file, harness/lib/stdio-bridge.mjs in the experiment repository, that forwards each message unchanged
What we changedA research build: five commits on the plugin (research v1, the benchmarked build), then two more after the benchmark (v2 and v2.1), transpiled into a copy of the installed package

Agents ran through mcp-audit agent, our agent harness, with built-in tools off and only the MCP tools available: two AI models, model A throughout and model B on set A. Every result was graded by a checker that read the database through REST, never by the agent’s own account.

Everything ran on loopback only, against throwaway sandboxes with a deterministic seed; no hosted service and no one else’s data was touched. Payload has no vendor-hosted MCP service: the plugin runs inside the customer’s own Payload app, so a pinned app and config fix what an agent sees. Token figures come in two kinds: benchmark tokens and cost are provider-reported, and sizes of what the server returns are tokenizer estimates of the text.

Headline finding: update and delete write through document locks (PAY-01, PAY-02)⁠

Payload writes a lock row when a user opens a document in the admin panel and refreshes it while the form is open; by default a lock expires after 300 seconds without a refresh. Its REST endpoints pass overrideLock as false, so another user’s update or delete gets HTTP 423. The plugin’s update tool defaults overrideLock to true, and its delete tool never passes the flag, so the Local API’s default of true applies.

Arm3.93.0v4 main ae5e1dc6, live
updatePosts while editor-b holds the lockWritten 10 of 10; editor-b’s lock row deleted 10 of 10; “success” text, no isErrorSame: written 10 of 10, lock deleted 10 of 10, isError 0 of 10
The same with overrideLock: falseRefused 3 of 3, as plain text without isErrorRefused 3 of 3, now with isError
deletePosts while editor-b holds the lockDeleted 10 of 10, success textDeleted 10 of 10; overrideLock refused by the schema
Control: REST update and delete as the same user423, 10 of 10 each; lock kept423, 10 of 10
Research buildWritten 0 of 10, deleted 0 of 10; each answer an isError conflict naming editor-bNot built
Our upstream-shaped 3.x patch, scripted replayWritten 0 of 10 with the lock kept (the refusal is still plain text on 3.x); deleted 0 of 10On main, the patch’s tests fail 4 of 4 before it and pass 4 of 4 after
Figure 1: Payload, writing to a post another user has locked

MCP update tool, vendor build

0 refused, 10 went through (10)

MCP delete tool, vendor build

0 refused, 10 went through (10)

REST update (control)

10 refused, 0 went through (10)

REST delete (control)

10 refused, 0 went through (10)

MCP update tool, research build

10 refused, 0 went through (10)

MCP delete tool, research build

10 refused, 0 went through (10)

Source: the headline table above (probes E1 and E2, and their re-runs on the research build).

An unsafe default, not a broken lock. Payload’s documentation says a locked document refuses updates and deletes from other users, and it documents that the Local API, which trusted server code uses, defaults overrideLock to true. The plugin passes that default on, so an agent acting for a signed-in user writes like trusted server code rather than like another user editing through REST. A refuter agent narrowed PAY-01 to exactly that, “an unsafe default with an opt-out”, because the update tool does accept overrideLock: false, and our Discussion asks the maintainers about the default. PAY-02 is stronger: the delete tool has no opt-out. Either way the write also deletes the editor’s lock row.

  • What it does to an agent. In the native session (job U8, the opening story), the agent’s stale write went through editor-b’s lock and dropped her saved row, and the agent told the user it was done. On the vendor build U8 then failed 5 of 5 benchmark runs; on the research build it passed 5 of 5, each run refused and telling the user who was editing.
  • A proxy cannot fix all of it. A rule in front of an unchanged server that sets overrideLock: false on update stopped the write (written 0 of 10, lock kept). The delete cannot be fixed that way: the vendor delete tool has no argument to set, and it still deleted 10 of 10.
  • Fix. Default overrideLock to false for the MCP tools only, and let delete take it; the Local API keeps its default. Raised in Discussion 19597, with the pull request prepared.

Second finding: match-all filters and partial bulk failures (PAY-03, PAY-05)⁠

InputThrough the tool, 3.93.0REST controlv4 main, liveResearch build
Update with a where of {}17 of 20 documents changed (our archive hook refused 3), 5 of 5 runs; a where of [] did the sameNo where: 400 “Missing ‘where’”, 5 of 517 of 20 changed, 5 of 5Refused 5 of 5 unless all: true
Delete with a where of {}All 20 deleted, 5 of 5 runs400, 5 of 5All 20 deleted, 5 of 5Refused 5 of 5
Bulk update, one of five documents refused by a hookHeadline “Multiple documents updated”, isError unset, 4 committed: 5 of 5400 with errors; the same 4 committed: 5 of 5Fixed: flagged 5 of 5PARTIAL headline with isError, 5 of 5
Bulk delete, one of five refusedSuccess text with “Errors: 1”, no isError (reproduced by hand before filing, 2 of 2)400 with errorsStill unflagged: 4 deleted, isError 0 of 5PARTIAL headline with isError by design; not probed

Why {} matters. In Payload’s own Local API, {} is a valid “match everything” filter, so the refuter narrowed this one too: the matching is not wrong; the gap is a missing guard on a destructive match-all write offered to an agent. The REST control is the closest request REST allows, not an identical one: REST cannot express an empty filter, and it refuses a missing one with HTTP 400. The guard is a behavior change, so we will ask in a Discussion before offering a patch.

Why the bulk answer matters. The four committed documents are the same on both surfaces; only the report differs. An agent reads the answer, not the database, and the headline says success. Pull request 19592 sets isError on the bulk delete on main, where the bulk update is already fixed.

MCP layer or platform? The control arm⁠

What we testedThrough the MCP serverThrough Payload’s REST APIWhere the problem is
Update a post another user has lockedWritten 10 of 10, lock deleted 10 of 10, success textRefused with HTTP 423, 10 of 10; lock keptMCP plugin: the update tool defaults overrideLock to true
Delete a locked postDeleted 10 of 10Refused with HTTP 423, 10 of 10MCP plugin: delete never passes the flag
Update and delete with a where of {}17 of 20 changed; all 20 deleted (5 of 5 runs each)No where (REST cannot send an empty one): 400 “Missing ‘where’”, 5 of 5MCP plugin: the guard only rejects an empty value
Bulk update with one failing documentSuccess headline, no isError, 5 of 5 (4 committed)400 with errors, 5 of 5 (the same 4 committed)MCP plugin: how the result is reported
Write fields named locale and draftNot stored, success text, 5 of 5Stored, 5 of 5MCP plugin, 3.x only
Edit an array row in a localized postOther-locale labels lost 5 of 5; row ids refused by the schemaKept 5 of 5 (REST sends row ids)MCP plugin; known issue 19435
Two simultaneous partial updates to different fields of one postOne field lost, 10 of 10 pairs (MCP plus REST)One field lost, 10 of 10 pairs (REST plus REST)Platform
Write from a stale readAnother editor’s row lost with no signal, 10 of 10 benchmark runs (A4 and U9)No version check on any surface; on v4 main the stale write lost the row 5 of 5, REST the samePlatform

The rows are from scripted probes with no model in the loop, each with its REST control, except the stale-write row, which is from the benchmark and the v4 re-run. Where the two surfaces accept different inputs, the control sends the closest REST request and the table says so: REST cannot send an empty where.

What is platform behavior. Two partial updates to different fields of one post, released together, lost one field in 10 of 10 pairs, and two REST updates lost one in 10 of 10 as well. By our house rule that is not an MCP finding, and we report it only so the MCP findings are not overstated. Payload has no version check that a client can send on an update, on any surface.

Payload: the findings⁠

This article reports 13 findings, PAY-01 to PAY-13; the experiment repository lists the same ones as F1 to F13. Each went through seven recorded checks: a reproduction with a raw JSON-RPC client and none of our tooling; a check against upstream main; the code’s history; Payload’s documentation, for intent; a search of prior reports; a refuter agent in a clean context (model A, a two-stage brief, code only); and a classification with a severity. For PAY-01 to PAY-10 those are 69 recorded results on the probes’ hash-chained log. PAY-11 to PAY-13 came from the completeness pass’s schema fuzzing (79 cases over the 9 write tools) and were recorded on that pass’s own log, with one refuter agent.

The v4 column is a live re-run on a sandbox built from main ae5e1dc6 for PAY-01 to PAY-10, each experiment with a REST control. For PAY-11 to PAY-13 it is a reading of the source at main f4662dd, marked “source”, not a live run.

The Severity column is the audit’s original scale: our judgment of operational impact, not a security rating. S2: real harm in normal use. S3: harm in a common case, or a gap with a workaround. S4: lower still; here, a correct refusal that drops its reason. Info: a cost or hygiene item. On that original scale no finding here is rated S1, the highest. The Rubric column rates each finding on IsoKron’s severity rubric v1.0, published after this audit; on the rubric, PAY-03 is Critical: one ordinary call deleted every document in the collection.

IDFindingMeasured on 3.93.0REST controlv4 mainRefuterClassSeverityRubricUpstream
PAY-01The update tool writes through another user’s lock and deletes the lock rowWritten 10 of 10, lock deleted 10 of 10; with overrideLock: false, refused 3 of 3 as plain text without isError423, 10 of 10Reproduced (10 of 10); the opt-out refusal now sets isErrorNarrowed: unsafe default with an opt-outUnsafe default: the MCP tools inherit the Local API’s default for trusted server codeS2High19591; Discussion 19597
PAY-02The delete tool deletes a locked document and has no overrideLock argumentDeleted 10 of 10423, 10 of 10Reproduced (10 of 10)ConfirmedBugS2High19591; Discussion 19597
PAY-03Update and delete with a where of {} match every document17 of 20 changed (3 refused by our hook); 20 of 20 deletedNo where: 400, 5 of 5Reproduced (5 of 5 each)Narrowed: {} is match-all in the Local APIDesign gapS2CriticalA Discussion, not yet filed
PAY-04Fields named locale, draft or select (and other names the tools use as arguments) cannot be writtenNot stored, success text, 5 of 5; the native job stored the market as nullStored 5 of 5Fixed by the data argument; confirmed 5 of 5Rejected: fixed on mainBug on 3.x; a backport candidateS3HighThe 3.x question in 19591; a 3.x patch is ready
PAY-05Bulk update and delete report success with no isError when some documents fail5 of 5400 with errorsBulk update fixed; bulk delete still unflagged (isError 0 of 5)NarrowedBugS3Low19591; pull request 19592
PAY-06Array edits lose other locales’ row values5 of 5Kept 5 of 5Row ids now accepted; with ids the other locale was kept 5 of 5Rejected: duplicate of 19435Cited, not claimedS3HighComment on 19435
PAY-07On 3.x, single-document writes echo the whole stored document3,278 echo tokens over 13 native jobs; a 21-byte update answered with 384 bytesREST can trim with selectNarrowed: only delete by id still echoes (836 bytes); update and create answer ids onlyNarrowedDesign gapInfoInformationalNot filed
PAY-08No tool annotations on 3.x0 of 12 tools annotatedNoneFixed (pull request 17027); 13 of 13 annotatedRejected: fixed on mainHygiene on 3.xInfoInformationalThe 3.x question in 19591
PAY-09Stale writes are accepted with no signal, on every surfaceNative agent lost another editor’s row; no signalSame: no version check on any surfaceReproduced (row lost 5 of 5, success text)NoneDesign gap in Payload’s core; the MCP layer can mitigate it with a best-effort checkS3High, platform-inheritedNot filed
PAY-10Two simultaneous partial updates to different fields lose one10 of 10 pairs10 of 10 pairs (REST plus REST)Reproduced (core)NonePlatform behavior, not an MCP findingInfoHigh, platform-inheritedNot filed (platform)
PAY-11Argument names the advertised schema forbids are stripped without a word; a misspelled-only update answers success and changes nothing4 of 79 fuzz casesREST also ignores unknown keys, but advertises no schemaSource: fixed (strict inputs)Rejected: fixed on mainBug on 3.x; a backport candidateS3LowNot filed; follows the 3.x answer
PAY-12Tool schemas omit the fields’ min, max, maxLength and configured locales, so schema-valid values are refused9 of 79 fuzz casesREST refused 8 of the 9 with a reason; it stored 1 (see below)Source: still present; main returns the reason and the schema after a refusalNarrowed: a design gap, 6 toolsDesign gapInfoLowNot filed
PAY-13A validation refusal names the field but drops the reason REST gives, with no isError12 fuzz casesREST gives the reason (“9 is greater than the max allowed Value of 5.”)Source: fixedRejected: fixed on mainBug on 3.x; a backport candidateS4LowNot filed; follows the 3.x answer

About the Rubric column: it rates what a failure did to the data. Whose code caused it, and any input the schema or the API would refuse, are noted below and do not lower the level. A loss counts as recoverable only when the audit tested a restore, and exposures are listed with their facts, not rated.

  • PAY-02 (High): No restore path found: in the audit’s delete check on the posts collection, which keeps versions, a delete also removed the post’s versions (2 to 0).
  • PAY-03 (Critical): One call removed documents it did not name one by one. The original S2 reflects the refuter’s narrowing, that {} is a valid match-all in Payload’s Local API; the rubric rates what the call did to the data and records whose behavior it is as attribution. MCP-specific: REST, the API Payload offers to clients, cannot send an empty filter and refuses a missing one. No restore path found: the tasks collection this probe used had Payload’s defaults, with no trash and no versions.
  • PAY-06 (High): Restore path documented, not tested: the posts collection keeps versions, but we did not restore from them. Medium if a restore is confirmed.
  • PAY-09 (High, platform-inherited): No surface of Payload has a version check, and REST lost the row the same way.
  • PAY-10 (High, platform-inherited): Two REST updates lost a field in 10 of 10 pairs as well. “Not an MCP finding” is the attribution; the level is the impact.
  • PAY-11 (Low): Reachability: needs argument names the advertised schema forbids.
  • PAY-12 (Low): Low, not Informational: in one of the nine cases the refusal named missing row labels instead of the locale, a false error message.

By class, that is 5 bugs (3 of them on 3.x only, fixed on v4 main), 1 unsafe default, 4 design gaps, 1 hygiene item, 1 known upstream issue cited and 1 platform behavior. In one of PAY-12’s nine cases REST stored the value, and the MCP refusal named missing row labels rather than the locale; in the other eight REST refused the value and gave the reason the MCP answer drops.

Static analysis also produced candidates about the plugin’s dev-only experimental tools, which register only when enabled in the plugin config under NODE_ENV=development. They are outside this article and not filed: v4 removed the source-writing tool families, and fixes for dev-only tools on 3.x are unlikely to be wanted.

How the findings were checked⁠

StagePayload
CandidatesDesk analysis of the pinned source (9 candidates on the per-collection find, create, update and delete tools, plus candidates on the dev-only tools outside this article), the native session’s friction register, and later schema fuzzing (79 cases over the 9 write tools)
Frozen probes on 3.93.0, each with a REST control8 experiments, E1 to E8, with 3 to 10 repetitions per arm, and 5 of them re-run on the research build: 13 runs
Completeness probes25 completed runs: read-only sweep, database statement trace, schema fuzzing, create-read-update-delete round trips, error grading, failure injection, key scope and prompt injection, and fixed call sequences
Re-run on v4 mainPAY-01 to PAY-10 live on ae5e1dc6, with a raw client and a REST control: PAY-01, PAY-02, PAY-03, PAY-09 and PAY-10 reproduced; PAY-05 half fixed; PAY-04 and PAY-08 confirmed fixed; PAY-06 and PAY-07 changed. PAY-11 to PAY-13 checked by reading the source at main f4662dd
Recorded checksAll 13 findings: the seven checks above. PAY-01 to PAY-10: 69 recorded results on the probes’ log. PAY-11 to PAY-13: on the completeness pass’s log
RefutersFresh agents in clean contexts (model A, the model that also ran the agents), code only: PAY-02 confirmed; PAY-01, PAY-03, PAY-05, PAY-07 and PAY-12 narrowed; PAY-04, PAY-06, PAY-08, PAY-11 and PAY-13 rejected as fixed on main or a duplicate; PAY-09 and PAY-10 have no refuter verdict
Register13 findings: 5 bugs, 1 unsafe default, 4 design gaps, 1 hygiene item, 1 cited upstream issue, 1 platform behavior
Hand reproductionThe reproductions run by hand on the audit’s sandbox and, at the same time, on a separate machine; sign-offs recorded on the probe logs
Reported upstream1 issue, 1 pull request, 1 Discussion and 1 comment on an existing issue, on 13 November
Agent series144 measured benchmark runs; 52 control runs (auditing on with per-run anchors, forced compaction, prompt injection) and 8 fault runs; 41 agent records comparing research v2 with v1 (stopped early by a billing refusal); 60 runs comparing research v2.1 with v1
Figure 2: How the Payload findings were checked
  1. FindDesk analysis, the native session and schema fuzzing on the pinned 3.93.0 build
  2. Probe with a control8 frozen experiments and 25 completeness runs, each against the same operation over REST
  3. Check and refuteRaw-client reproduction, v4 checks and refuter agents in clean contexts; 13 findings registered, including those narrowed, fixed on main or already known upstream
  4. Reproduce by handOn the audit’s sandbox and on a separate machine, with sign-offs recorded
  5. File1 issue, 1 pull request, 1 Discussion and 1 comment on 13 November

Source: the stage table above.

What the checking caught, in our own records. The recorded classification text for PAY-01 says 9 of 9: it was drafted from a dry run before the frozen run finished, and the frozen run recorded 10 of 10. The log keeps the text as written; the audit report records the correction. Checked against the per-run table before this article, three hand-written figures were also corrected: the false “done” count, and the model B and observer totals, which had included A3. The figures below are the corrected ones. The v4 re-run also showed that one main-branch code reading was wrong: on main, update and create no longer echo the document (PAY-07 narrowed to delete).

What held up⁠

  • Access rules. The plugin passes the key’s user with overrideAccess: false, so collection access rules apply. A key scoped to three tools listed exactly those three; calls to other tools were refused with nothing changed (4 of 4); a capability switched off mid-session was refused on the next call; a disabled, wrong or missing key got 401. The one place MCP let a user do more than REST is the document lock.
  • Partial updates. A partial update never cleared the fields it did not name, on either build or over REST.
  • Transactions. A server killed mid-write rolled back: 0 of 3 writes landed, on both builds and in the controls.
  • Reads. Every find answer’s Total matched REST, and no find answer contained an e-mail address. A list read costs 4 database statements against REST’s 3, with no fan-out and no repeated statements.
  • Retries before a write. After an injected 504 in the native session, and after 429, 401 and 503 errors in the fault runs (6 of 6), the agent re-read state before retrying and never duplicated.
  • Agent reports. Most vendor-build failures in the benchmark were honest or unclear, not false successes (the false “done” labels are discussed under the benchmark).

Usability pass: what the server costs an agent⁠

Before designing any change, model A used the vendor build for 13 everyday jobs, one run each on a fresh seed, with only the 12 MCP tools: no shell, no files, and no mention of an audit. A Phase 1 baseline run came first.

SessionJobs passedJob tool callsResponse tokensCost
13 everyday jobs, one run each9 of 134411,4340.62 USD (14 runs, the baseline run included)

It failed 4: U2 (the localized row labels were lost, issue 19435), U6 (the market field locale was stored as null; the agent noticed and told the user), U8 (the opening story: a stale write through the lock, success claimed) and U9 (a stale array write dropped another editor’s row). After an injected 504 in U7, the agent checked before retrying and did not duplicate. U3 and U13 passed because the agent found a refused document in plain error text that carried no isError.

Of the 11,434 response tokens, write echoes were 3,278; the other 8,156 were find results and short answers, which our toolkit counts as fully needed because it cannot split Payload’s prose-plus-JSON answers, so this split is coarse. Each session start cost 4,195 tokens before the first call, 4,149 of them for the tool list.

Figure 3: What the server returned in the native session, by part (cl100k tokens)

13 everyday jobs, 44 tool calls

3278 write echo, 8156 everything else (11434)

Source: the session’s response tokens, as split in the paragraph above.

The friction register has 9 items: the lock takeover, the stale write, the localized rows (the known issue), the colliding field names, write echoes, catalog weight, missing isError, no instructions or annotations, and writes one document at a time. The change list has 7 items, all built into the research build; catalog weight was measured and held: the research build’s new arguments add about a thousand tokens, and removing the schema each tool repeats would need shared schema definitions that the MCP SDK the plugin uses does not support.

The Payload research build⁠

Five commits on the pinned plugin, containing every change from the session plus the bug fixes:

  • Locks respected by default, as on REST: overrideLock defaults to false, and delete accepts it. A locked write returns an isError conflict that names the holder and gives the lock’s age, the document’s updatedAt and the current document, with the instruction not to take over unless the user asked.
  • A best-effort version check: an optional ifUpdatedAt, plus the version this user last read as an implicit check. A stale write gets a conflict carrying the current document. This is read, then write, in two steps, not an atomic compare-and-set; that needs support in Payload’s core.
  • Array rows keep an optional id, equivalent to pull request 19436.
  • A data argument for fields whose names collide with tool arguments (additive; nothing renamed).
  • Compact write receipts (id, updatedAt, status, fields written); returnDocument: true gives the old full answer.
  • isError on every error, and bulk answers with one result per document under a PARTIAL headline.
  • Six lines of default server instructions and tool annotations.
  • A where of {} is refused unless all: true is passed.

The research catalog is 5,060 cl100k tokens against 4,031, 25% more. Re-run on the research build, the lock probes wrote 0 of 10 and deleted 0 of 10, each answer an isError conflict naming editor-b (10 of 10); the empty-filter update and delete were refused 5 of 5; the partial bulk update answered PARTIAL with isError 5 of 5.

After the benchmark the build was also type-checked and tested inside Payload’s monorepo: 0 type errors in a strict build. Payload’s own plugin integration suite passes 46 of 74 as is; all 28 failures assert the old full-document write text that the compact receipts replace by design, and with returnDocument: true in those calls it passes 74 of 74.

Benchmark: vendor build against research build⁠

We ran 144 measured runs on 13 November, from 07:29 to 08:16 UTC, in a seeded shuffled order alternating builds, each from a fresh seed. Model A ran 5 runs per task and build on set A (60 runs, A3 included) and set B (50). Model B ran 2 per task and build on set A (24, A3 included). An observer arm with our capture proxy off ran 1 per task and build on set A (10, A3 included, A4 excluded). The tables regroup them: set A’s five pre-registered tasks (58 runs, model A and the observer), set B with A3 (62 runs), and model B’s 20 runs on the pre-registered tasks, in the text; model B’s 4 A3 runs passed on both builds and are in no table. The benchmark cost 5.44 USD in model spend. Correctness is a checker reading the database through REST after each run, never the agent’s account.

First, the limits. Probe counts such as 10 of 10 show that a behavior occurs under the stated conditions, not how often. The benchmark runs each task 5 or 6 times per build, which shows spread, not a rate, and the two builds were measured as wholes, with no fixes-only arm. Completion intervals are 95% Wilson score intervals with runs pooled; intervals for differences in tokens and cost come from a task-clustered bootstrap (10,000 resamples of tasks, seed 1). The full list is under Benchmark limits.

Set A: tasks fixed before any change was designed⁠

Set A’s prompts and checkers were committed at 06:18 UTC on 13 November, before any research-build code was written, and fixed by the pre-registration freeze at 06:43 UTC, on the evidence chain of the native-session series. A4’s seed and description changed with our toolkit’s conflict rule before that freeze and before any agent run. The benchmark series itself was frozen at 07:28 UTC, after the change list was written; the measured runs followed. The fair test is five tasks: A1, A2, A4, A5 and A6 (A3 is explained below).

TaskVendor buildResearch buildWhat decided it
A1 Publish a post and add a task5 of 6 (one credit error)6 of 6None
A2 Change one array row’s link0 of 66 of 6Other-locale labels lost on the vendor build (issue 19435); kept with row ids
A4 Add a row; editor-b adds hers after the agent’s read0 of 55 of 5Vendor: the stale write drops her row. Research: a conflict, then an informed retry
A5 Delete one person’s done tasks6 of 65 of 6 (one credit error)None
A6 Add a listing for the de-DE market0 of 66 of 6Vendor: the locale field cannot be written. Research: data
Total11 of 29 (38%, 95% interval 23% to 56%)28 of 29 (97%, 83% to 99%)None
Tokens over all attempts984,6421,159,007None
Expected tokens per verified success89,51341,393None

Two runs, one per build, failed with an API credit error on our account. Both totals count them as failed attempts with zero tokens; they were not graded.

Figure 4: Payload set A, runs passing per task (model A and the observer arm)

A1 Publish a post and add a task

Vendor build5
Research build6

A2 Change one array row’s link

Vendor build0
Research build6

A4 Add a row after another editor’s change

Vendor build0
Research build5

A5 Delete one person’s done tasks

Vendor build6
Research build5

A6 Add a listing for one market

Vendor build0
Research build6

Each row is drawn to its own scale.

Source: the set A table above.

Per task, model A and the observer arm (medians from our toolkit’s report, which counts the two credit-error runs as zero-token attempts; leaving them out does not change the token and cost conclusions below):

TaskTool calls, vendor → researchTokens per run, vendor → researchCost per run, USD, vendor → research
A1 Publish a post and add a task4 → 435,385 → 41,2400.0441 → 0.0439
A2 Change one array row’s link2 → 234,373 → 40,1060.0337 → 0.0344
A4 Add a row after another editor’s change2 → 334,294 → 55,8160.0329 → 0.0504
A5 Delete one person’s done tasks3 → 336,980 → 40,7570.0374 → 0.0366
A6 Add a listing for one market2 → 133,659 → 26,3420.0338 → 0.0281

The research build costs more where it adds work (A4’s conflict and retry) and less where data lets one call do the job (A6); the catalog it adds is re-read from the prompt cache on every turn.

“Expected tokens per verified success” is the tokens of every completed attempt divided by the runs a checker verified: 984,642 over 11 on the vendor build and 1,159,007 over 28 on the research build. It roughly halved because far more runs succeeded; tokens per run went up. The median run used 34,373 tokens on the vendor build and 40,709 on the research build, a difference of 6,336 with an interval of −7,445 to 21,477. Mean cost per run moved from 0.0351 to 0.0376 USD; the interval for that difference, −0.0037 to 0.0102, includes zero. The audit attributes the rise to the larger catalog and instructions and to the extra calls of conflict recovery; that split was not measured separately.

With five tasks, a per-task sign test cannot show much: 4 tasks improved and 1 got worse, p = 0.375, not significant. The one that got worse, A5, lost a run to the API credit error, as A1 did on the vendor build. At run level, McNemar’s exact test pairs each task’s vendor and research runs by run order: 18 pairs favored the research build, 1 the vendor build and 10 were tied, p = 0.0000763. It treats the 29 pairs as independent although they cluster in five tasks, so it describes these runs and does not show that the result carries over to other CMS work.

Model A alone on the five pre-registered tasks, without the observer arm, went from 9 of 25 to 24 of 25, with medians of 34,365 and 40,709 tokens and the same interval for the difference. Model B, 2 runs per task, went from 4 of 10 to 10 of 10 on the five pre-registered tasks, failing the same three on the vendor build (A2, A4, A6). The observer arm, with our capture proxy off and one run per task (A4 had none), showed the same pattern (2 of 4 against 4 of 4), and a later arm with auditing on and an anchor after every run gave it again (4 of 10 against 10 of 10, 2 runs per task), so the capture did not change the outcomes.

No run on either build chose a wrong tool (0 of 158 runs: the 144 benchmark runs and the 14 native-session runs). What failed on the vendor build was the first write’s arguments: for model A on the pre-registered tasks, the first write was valid on the first try in 21 of 28 runs against 28 of 28, mostly the field-name trap of A6 and row ids the vendor schema refuses. The vendor build’s failures were also silent: 17 model A runs and 6 model B runs on these tasks failed their checker with no refusal returned, against none on the research build.

Of the 17 graded vendor-build failures on the five pre-registered tasks, 8 were honest (the agent said it could not do the job), 7 were unclear and 2 were classed as false “done” claims, both on A6, where the market field could not be written. The label is mechanical. The job asked for “a finished offer”, and both answers called the listing “finished”, which our claim rule reads as a done claim. Both then told the user the saved record showed no market, so the agent did not hide the failure.

A3: a pre-registration defect, disclosed⁠

At the pre-registration freeze (06:43 UTC), A3’s prompt file was a copy of a different job’s prompt (“Every task that isn’t done yet should be priority 3 from now on…”), while A3’s frozen description and its checker were “mark every ‘doing’ task as done; say if any could not be changed”. The prompt did not match its own pass criterion. A dry run of the benchmark, not measured, caught it: the agent set priorities, and the checker failed it. After the pre-registration freeze, but before the benchmark freeze (07:28 UTC) and so before any measured run, we changed the prompt to match the frozen description: “Please mark every task that is currently ‘doing’ as done, and tell me if any of them couldn’t be changed.”

A task changed after its pre-registration freeze cannot count as pre-registered, and our toolkit refused it as one (“prompt changed since the freeze”). So A3 is reported with the friction-derived set below. It passed 6 of 6 on both builds and does not touch the set A totals above.

Set B: tasks from the session (favors the research build by construction)⁠

Set B is the session jobs that failed, plus a clean-up job and A3. These tasks were chosen after the changes were designed, from the session that produced them, so they favor the research build by construction.

TaskVendor buildResearch build
U2 Localized row edit0 of 55 of 5
U6 Three listings, market en-GB0 of 55 of 5
U8 Add a row while editor-b keeps the post open0 of 5 (lock taken over, her row lost)5 of 5 (lock respected, honest answer)
U9 Add a row after a stale read0 of 55 of 5
U11 Scratch clean-up5 of 55 of 5 (more calls: one delete per document; see the appendix)
A3 (moved here)6 of 66 of 6
Total11 of 31 (35%, 95% interval 21% to 53%)31 of 31 (100%, 89% to 100%)
Tokens over all attempts1,203,0091,619,130
Expected tokens per verified success109,36452,230

Write echoes shrank where writes dominate (A6: 419 to 341 response bytes per run; U2: 1,486 to 995) and grew where conflict answers carry the current document (U8: 1,563 to 5,360).

Conflict recovery⁠

Three tasks seed a conflict right after the agent’s first read: in A4 and U9 another editor adds a row and leaves; in U8 editor-b saves a row and keeps the post open, holding the lock to the end of the run. Our toolkit classifies each run from the agent’s own calls under its conflict rules. The recovery rate is runs that re-read (or used the current state the conflict answer carried), kept the other change and landed their own, out of the conflicts the build signaled. We report version 3 of those rules, re-applied on 13 November to the saved records of every run; the version 2 records made during the benchmark sit beside them.

The research build recovered 10 of the 15 seeded conflicts: A4 5 of 5 and U9 5 of 5, with a median of 1 extra call and about 60 extra response tokens per recovery. Model B recovered A4 2 of 2. The vendor build never returned a conflict: no signal in 15 of 15 seeded runs (2 of 2 for model B), and the other writer’s change was lost every time.

U8, the other 5. In all five research runs editor-b held her lock to the end; the agent’s writes were refused with a conflict naming her, it did not take over, and it told the user who was editing. Our checker passed all five. They are recorded as “gave up and reported the failure” (gave_up_reported_failure), which counts against the rate. The rules have a separate class, “conflict persisted”, for a run that ends with the other writer still holding the lock, and it sits outside the rate; our U8 checker was frozen before that class existed and could not report whose lock remained, so 0 runs were recorded in it. A later series with an updated checker recorded five new U8 runs on the same build as “conflict persisted”, 5 of 5. The appendix gives the detail; the benchmark’s recorded classes stand.

Figure 5: Payload, seeded conflicts in the benchmark, by outcome

Vendor build, 15 seeded

0 recovered, 0 refused, reported who was editing, 15 no signal, other change lost (15)

Research build, 15 seeded

10 recovered, 5 refused, reported who was editing, 0 no signal, other change lost (15)

Source: version 3 of our toolkit’s conflict rules, re-applied to the saved records, in the text.

Benchmark limits⁠

  • Five or six runs per task and build (two for model B); repetitions measure spread within one prompt, not variety across CMS work. Statistics with five tasks cannot show much, and the run-level test treats run pairs as independent.
  • Set A’s prompts were committed before any change was designed, but A4’s seed changed before the pre-registration freeze, and A3’s prompt after it, before any measured run (A3 moved to set B). Set B favors the research build by construction.
  • The research build is measured as a whole, with no fixes-only arm. We did not split the change by type: no ablation and no accounting model was run, so which change produced which gain is not measured, beyond the per-task reasons in the table.
  • Prompt caching carries across runs; the shuffled order exposes both builds equally. Two runs were lost to a billing error and count as failures.
  • One client, two models, one sandbox, agents running as one admin user. Other clients and pricing will give other numbers.

After the benchmark. The completeness pass found that research v1 made batching worse (U11 in one call in 0 of 10 opportunities, against 8 of 10 on the vendor build) and that its bulk calls had no version check. Research v2 fixed both but lost editor-b’s change in 6 of 9 conflict runs, and research v2.1 fixed that. In a final series, research v2.1 batched U11 in 10 of 10 opportunities and kept editor-b’s change in 15 of 15 conflict runs, as research v1 did. The benchmark figures above are for research v1. The appendix has the history.

Other checks from the completeness pass⁠

  • Retries after a failure. A retry is safe when the failure came before the write. After a write had landed, a blind retry of a create made a second copy (5 of 5 after a client timeout) unless a unique field refused it, and a delete delivered twice answered “Not Found” for a delete that had succeeded (5 of 5). Neither surface has an idempotency key. A real agent re-read state before retrying after 429, 401 and 503 errors and never duplicated (6 of 6), but could not see a duplicate the network made (0 of 2).
  • Prompt injection. Stored text addressed to the assistant came back verbatim and unmarked inside the find results on both builds. The agent acted on it in 0 of 12 runs, but never warned the user, and in 9 of 12 it left the passage out of its answer without saying so.
  • Fixed call sequences. Without a model, the best fixed sequence of calls passes 3 of 6 set A tasks on the vendor build and 6 of 6 on the research build.
  • Compaction. A forced mid-task compaction changed no pass pattern on the research build.
  • Spend. Model spend charged for the whole audit was about 11.58 USD; the last two series ran on a subscription plan, about 4.77 USD of estimated usage.

Upstream status⁠

The linked items below were filed on 13 November 2026, after we reproduced the filed cases by hand and read every patch. A minimal public reproduction of PAY-01, PAY-02 and PAY-05, on 3.93.0 and on 4.0.0-beta.9, is in isokron-ai/payload-mcp-repro.

FindingsUpstream itemStatus as of 15 November 2026
PAY-01, PAY-02, PAY-05Umbrella issue 19591: update and delete write through document locks, and partial bulk failures report success; asks whether the maintainers want 3.x fixesOpen; labeled Bug and plugin: mcp by Payload’s triage automation, no reply yet
PAY-05Pull request 19592: isError when a bulk delete fails or partly fails, on main, with tests that fail before and pass afterOpen, ready for review with its checks passing, awaiting maintainer review
PAY-01, PAY-02Discussion 19597: should the MCP update and delete tools respect document locks by default, like REST? A pull request with tests is prepared and waits for a reply there, or a weekOpened 13 November; no comments yet
PAY-06A measured reproduction on the known issue 19435, offering to test its fixPosted 13 November
PAY-03A Discussion on a guard for a match-all where in update and delete; a pull request only if the maintainers agreeNot yet filed; next in our filing order
PAY-04, PAY-08, PAY-11, PAY-13Backports from v4 to 3.x, only if the maintainers want 3.x fixes. Issue 19591 asks, naming PAY-04 and PAY-08; a 3.x patch for PAY-04 is ready, and PAY-11 and PAY-13 would follow the same answerWaiting on that answer
PAY-07, PAY-09, PAY-10, PAY-12Not filed: two design gaps, a possible version-check proposal for v4, and platform behaviorNone

As of 14 November 2026, no maintainer had replied to these filed items; the labels came from Payload’s triage automation. We file at most two new threads a day and keep at most two of our pull requests open and unreviewed. We will record maintainer responses and fixes in the changelog below. Maintainers are not obligated to review, accept or merge our submissions.

Reproduce it and rebuild the builds⁠

Reproduce the Payload findings. isokron-ai/payload-mcp-repro is Payload’s blank template with one posts collection, two admin users and the MCP plugin, on SQLite. After pnpm i && pnpm seed && pnpm dev, node scripts/repro-locks.mjs, scripts/repro-bulk-delete.mjs and scripts/repro-bulk-update.mjs run PAY-01, PAY-02 and PAY-05, each with its REST control; v4-beta/ is the same app on 4.0.0-beta.9. The full harness, with the Postgres sandbox, the probes, the tasks, the checkers and the stdio bridge, is in the experiment repository below.

Rebuild the builds. The patch series, the three research builds (v1, v2 and v2.1), the harness, the findings register and the benchmark reports are in isokron-ai/payload-mcp-writepath-experiment, with Payload’s MIT license and upstream’s plugin source at v3.93.0. Its harness/build/build.sh checks the upstream import and the patch series, and rebuilds the 7 changed files of each build byte-identical to the builds we measured. The research builds are experiments, not upstream releases: use Payload’s own plugin for real work.

Evidence⁠

Each Payload series was frozen before its runs (the harness, seed, checkers, prompts and toolkit commit hashed), recorded on a hash-chained log, and anchored off-box at its freeze, most of them also with RFC 3161 timestamps from two independent authorities. The hand sign-offs are recorded on the two probe logs. There are nine series. Seven are closed into Evidence Packs, and each verifies with our toolkit’s verify command: the native session (with the baseline run), the frozen probes, the benchmark, the completeness probes, the completeness agent arms, the fault re-runs and the research v2.1 agent series. The other two, the research v2 batching series and the v2.1 probes, are recorded and anchored but not yet closed. The packs are not published with this article, so readers cannot verify them today; How to verify an MCP audit report yourself shows what a pack contains and how one is checked.

What this does not show⁠

  • Small counts. 3 to 12 trials or pairs per scripted arm, and 5 runs per benchmark task and build (2 for model B). Deterministic code paths (10 of 10, 5 of 5) show a mechanism under the stated conditions, not how often it happens; only the simultaneous pairs are races.
  • No natural collisions. Every agent run was one agent alone, with the other writer seeded by our harness at a chosen moment. editor-b’s admin-panel save was simulated as a REST update of the items she saw when she opened the post, and her lock was written by our harness; the admin panel’s own handling of a stolen lock was not exercised.
  • Combined builds and favorable tasks. Each build was measured as a whole, with no fixes-only arm, and set B favors the research build by construction. The research build’s version check is read-then-write, not atomic. The benchmarked build is the transpiled research v1; v2.1 was measured only on six tasks in one later series. Both builds protect only writes made through them: a REST or admin-panel writer can still overwrite an agent’s change.
  • Refuters on the same model. The refuter agents ran on model A, the model that also ran the agents. Clean contexts keep a refuter from seeing our reasoning, but the same model can share the same blind spots; our GitHub audit used refuters on a different model.
  • One sandbox. Payload ran under next dev (both builds), on one Postgres; v4 was re-run live with a raw client only, with no agent runs and no research build, and PAY-11 to PAY-13 were checked on v4 by reading the source only. Payload keeps no request log, so its target-side evidence is our REST snapshots. One client, two models; other MCP clients may present the tools differently.
  • Evidence. Seven of the nine Payload evidence series are closed and verify, but the packs are not published, and two series are not yet closed (Evidence).

About this research⁠

IsoKron performs fixed-price MCP server audits with this method: probes against a pinned build with REST controls, a native agent usability pass, clean-context refutation, re-checks against upstream main, tested patches and filing upstream; the before/after benchmark is an optional, quoted add-on. Details: MCP server audits. We also build MCP servers and plugins to the rules in Send the change, not the object; we do not present a test of code we built as independent.

Appendix: the research builds after the benchmark⁠

The completeness pass found one thing research v1, the benchmarked build, made worse. On U11 (delete three scratch tasks and three temporary listings), model A handled the batch in one call in 8 of 10 opportunities on the vendor build and 0 of 10 on research v1, because research v1’s own instruction told agents to pass ifUpdatedAt, one document’s version, on every update and delete. Reading research v1’s code for the fix found a worse gap: its read-version check ran only on writes by id, so a bulk call by where had no precondition. Probes showed research v1’s bulk calls deleting or overwriting a document changed after the read in every valid repetition (5 of 5 for delete, 4 of 4 for update).

Research v2 changed the instruction (several documents in one call with where; every update and delete, bulk included, checked per document against the version last read) and added the check to bulk calls. Batching came back (U11 8 of 8 opportunities, A5 5 of 5), but research v2 lost editor-b’s change in 6 of 9 conflict-guard runs. Each of the six first read the post with a select that left out createdAt, which the memory of the version last read is keyed on, so the write had no precondition. Research v1 had the same gap, hidden because its agents passed ifUpdatedAt explicitly. Research v2.1 keeps that memory through a narrow select: in a probe of that read, the stale write landed 5 of 5 times on research v2 and was refused 5 of 5 on research v2.1.

In a final series that night (model A, 5 runs per task and build, 60 of 60 completed), research v2.1 batched U11 in 10 of 10 opportunities and A5 in 5 of 5 (research v1: 0 of 10 and 0 of 5). It kept editor-b’s change in 15 of 15 conflict-guard runs on A4, U9 and U8, and research v1 did too in this series (15 of 15). Every research v2.1 first write was refused by the server’s read-version check with no ifUpdatedAt from the agent, including 7 runs that first read with the narrow select behind research v2’s losses. Batching cut calls (U11 from 8 to 4, A5 from 3 to 2) but not tokens per run, because research v2.1’s larger session start is re-read on every turn. These are small cells with one model, reported as counts. The benchmark figures in the body are for research v1; research v2.1 was not run on the full task sets.

Appendix: how the U8 runs were classified⁠

Version 3 of our conflict rules counts a run as “conflict persisted”, outside the rate, only when the task’s checker reports that the other writer’s own lock was still in place at the end. Our U8 checker was frozen before version 3 existed and recorded only how many lock rows remained (one in each research run), not whose. We tried to fill the field afterwards from the saved end state, but editor-b’s identity is not in it, so the field stays empty and the five runs count as gave_up_reported_failure, inside the rate.

The saved state does suggest what happened: in every research U8 run exactly one lock row remained, the only lock rows created in the runs came from editor-b’s setup, and the server’s last conflict answer, seconds before the end, named her as the holder. That makes it very likely these were persisted conflicts handled correctly, but that is an inference, not a measurement, and we do not report the benchmark’s rate as 10 of 10 with 5 persisted. The U8 checker was changed after the benchmark freeze to record lock owners. In a later series the same night, with that checker, five new U8 runs on the benchmarked build were recorded as conflict_persisted in 5 of 5: refused three times while editor-b held her lock, nothing forced, the user told the post was being edited. That supports the inference; it does not change the benchmark’s recorded classes.

Changelog⁠

  • 14 November 2026: published.
  • 14 November 2026: added IsoKron rubric v1.0 ratings beside the original labels: rating lines under the headline and second findings, a Rubric column in the findings table, and notes on the ratings after it. The sentence that no finding is rated S1 now says it refers to the original scale; on the rubric PAY-03 is Critical.

Get new research by email

New MCP audits and research, about twice a month, plus occasional IsoKron service news. Unsubscribe anytime.

Sent with Kit. See our privacy page.

How this was tested. AI agents run much of the testing: scripted checks, attempts to disprove each finding, and real tasks that use the server the way an agent would. I also test every server myself on a separate machine and check each finding against the AI results. A finding appears here only if it holds up in both. — Joshua Bryan

For this piece: Findings were found with AI assistance, reproduced in scripted probe runs with no model in the loop, each against the same operation sent to Payload's REST API, and challenged by refuter agents in clean contexts (on the model that also ran the agents) that read the code only. PAY-01 to PAY-10 were re-run live on Payload v4 main; PAY-11 to PAY-13 were checked against v4 main's source. The reproductions were also run by hand, on the audit's sandbox and at the same time on a separate machine. Seven of the nine evidence series are closed into Evidence Packs, and each verifies; the two batching series are not yet closed. Benchmark correctness is an independent checker's reading of the database through REST, never the agent's account.

View as Markdown (for agents and scripts).