Grafana MCP server v2.2.0: dashboard patches skip the version check and can silently lose edits

Self-funded research by IsoKron, conducted and written with AI assistance. Not commissioned or endorsed by Grafana Labs; IsoKron is not affiliated with Grafana. IsoKron sells MCP server audits and maintains the tools used here.

We built Grafana's official MCP server, mcp-grafana version 2.2.0, from a pinned commit and tested its write tools against two local Grafana instances seeded with throwaway data: Grafana 13.3.0, which serves dashboards through the newer Kubernetes-style API, and Grafana 11.6.16, which uses the legacy API. The audit combined a static pass over the write-side code, 26 scripted probe runs with controls through Grafana's own HTTP API, and a before/after benchmark of routine observability tasks run by a Claude agent.

The most serious finding concerns update_dashboard in patch mode. The tool saves every patch with Overwrite: true hard-coded, so the version check in Grafana's save API never runs. Two patch calls released together from two server processes lost one of the two edits in 30 of 30 pairs on both Grafana versions; both calls returned success. The control is not clean: two simultaneous saves sent straight to Grafana's API with overwrite off also lost an edit, in 30 of 30 pairs on Grafana 13 and 20 of 30 on Grafana 11. What the tool adds is the bypass: Grafana rejected a stale save, sent after the other writer's had landed, in 10 of 10 trials on both versions. The lost edit stays in the dashboard's version history, and only classic dashboards are affected. The proposed fix, as it now stands in pull request 1518, lost no edit in 60 simultaneous pairs on Grafana 13, where it saves through the Kubernetes-style API. On Grafana 11, which has only the legacy save endpoint, it reports stale saves as conflicts but still lost 14 of 30 exactly simultaneous pairs.

A second tool, the alert-rule update, rebuilt the whole rule from the call's arguments: in 3 of 3 trials on each version, an update that left out the optional fields reset the rule's labels, annotations, paused state and notification settings while reporting success.

The static pass produced twenty candidates; a refuter upheld fourteen, and six became fixes. Six usability changes came from friction the benchmark agents reported. All twelve are proposed upstream in four open pull requests, none yet reviewed by a maintainer. While preparing those for review we added eight further changes, listed under Rebuild the fixed server; the benchmark did not measure them. In the benchmark the fixed build made 15% fewer MCP tool calls (225 against 264 over thirty runs), concentrated in calls the usability changes target; the benchmark does not measure the six fixes. The two security candidates we started with were dropped during triage: both turned out to be documented, intended Grafana behaviour, not reportable vulnerabilities.

Setup and boundary#

Everything ran against two local Grafana instances IsoKron operates, seeded with fake dashboards, datasources, a contact point, alert rules and a silence; no other users' data was touched.

Not tested: prompt injection, load, a second Grafana organization, any hosted Grafana, and tools whose backends the stack lacked (Grafana Incident, Mimir or Cortex rulers, Agent Observability) or that the Editor token cannot use (datasource writes).

Headline finding: patch-mode dashboard saves bypass Grafana's version check (GF-01)#

Patch mode reads the dashboard, applies the operations and writes the whole dashboard back. In v2.2.0 that write sets Overwrite: true unconditionally (tools/dashboard.go) and ignores the caller's overwrite argument, so a save another writer made in between is replaced.

A truly simultaneous race is not specific to the MCP tool: given two saves carrying the same version with overwrite off, Grafana's own API accepted both and lost an edit in every pair on Grafana 13 and in two of every three on Grafana 11. What is specific to the tool is the forced overwrite: Grafana rejects a save made from an outdated version once the other save has landed, and the tool switches that check off for every patch. Patch mode re-reads inside the call, so a stale save cannot be staged through it; its exposure is the gap between its own read and save (tens of milliseconds in the harness), which the first row measures. The stale-save arm was added after a dry run and was not part of the pre-registered verdict. Both original controls used POST /api/dashboards/db, the endpoint the tool also saves classic dashboards through. The Kubernetes-style API was raced separately on 10 November, on Grafana 13 only, and rejected one of the two saves in every pair.

• Limits: only a save landing inside that gap is lost, and the lost edit can be restored from the dashboard's version history. Classic dashboards only; the v2 patch path sends the resource version it read. The pairs were deliberately simultaneous; how often this happens in normal use was not measured.
• Proposed fix (pull request 1518): a patch save is checked against what was read. Where the dashboard was read through the Kubernetes-style API (Grafana 12 and later), it is written back through that API with the resource version that was read. Where it was read through the legacy API, it is saved with the version that was read and without overwrite. A save that loses returns a conflict, and sending the patch again applies it to a fresh read. Unit tests cover both conflict paths; the integration tests compile but were not run.
• Re-runs on 10 November, 30 simultaneous pairs per round. The build used in the benchmark has the version check only, and lost between 11 and 21 of 30 per round on both Grafana versions: the pairs where the legacy save endpoint accepted both saves. Pull request 1518 as now proposed lost 0 of 60 on Grafana 13; every pair returned one conflict, and a single retry landed both edits in all 60. On Grafana 11, which has only the legacy endpoint, it lost 14 of 30, so there the fix gives the tool Grafana's check and no more. In 2 of the 15 conflicts on Grafana 11 the single retry was itself rejected; no edit was lost and we did not retry again. These re-runs were made after the audit's evidence chain was closed and are not part of it.
• Related gap on full-body v2 saves (GF-19, in the register below): the same pull request adds an optional expectedVersion.

Two agents, or an agent and a person, saving the same dashboard at once is plausible, and nothing reports the loss. For teams letting agents manage dashboards, this is the finding we would act on first.

Second finding: partial alert-rule updates silently reset omitted fields (GF-03)#

The alerting_rules_write update operation builds the rule from the call's arguments and PUTs it, with no read of the stored rule. We created a rule with labels, annotations, a paused state, a notification receiver and keep_firing_for, then sent an update carrying only the required fields.

• Grafana 13: all five were reset, in 3 of 3 trials.
• Grafana 11: the first four were reset in 3 of 3 trials; that version does not return keep_firing_for.
• Controls, both versions: a direct PUT of the full rule to Grafana's API, and a tool update with every optional field supplied, each lost nothing in 3 of 3.

Grafana's PUT is a full replace and the tool keeps whatever it is given; the loss comes from the tool not reading the stored rule first. We did not send a partial PUT straight to Grafana's API. The v2.2.0 tool description says update "replaces the whole rule: it takes all required fields", so full replace is the stated design, but it does not say optional fields are cleared, and the schema marks them optional.

The proposed fix (pull request 1519) fetches the existing rule and overlays only the optional fields the caller supplied. Required fields must still be sent: the fixed build's tool description says they are preserved when omitted, but in one benchmark run three updates sent without title were each rejected with "title is required". The description or the code in that pull request needs correcting.

Findings register#

The static pass listed twenty candidates. The triage plan sent seven to live probes, left twelve to code reading (the two security candidates among them) and set one aside as intended behaviour. The outcomes add up to twenty:

Of the seven unfiled items, six were rated S4; GF-04 (an external Alertmanager/ruler read-modify-write, from code reading only) was rated S3. Only GF-15 was probed live.

IDs: GF-nn is a candidate from the static pass (GF-21 was added later, from the benchmark). A–D are working labels for the twelve changes; the usability ones are A1, A3, A4, B1, B2, D1.

Six fixes, each from an upheld candidate. The refuter classed GF-01 and GF-02 as reliability, GF-03, GF-08 and GF-19 as usability and GF-18 as hardening, and rated the first three S3 and the other three S4:

Six usability changes, from friction the benchmark agents reported, not refuter-checked as defects:

Upstream status, checked 10 November 2026: all four pull requests are open with review required, and none has been reviewed by a maintainer. The latest release is still v2.2.0. One further finding, GF-21, is documented next and not fixed.

What we left unfixed: Grafana 13 default materialization (GF-21)#

On Grafana 13, a panel-only patch changed five top-level dashboard fields the agent never touched, the five the checker compares (schemaVersion also rises from 39 to 42 on dashboards stored at schema 39). The same patch on Grafana 11 preserved them.

The cause is the read path. On Grafana 13 the server reads classic dashboards through the Kubernetes-style v1beta1 API, which returns the dashboard migrated to the latest schema with defaults filled in, and the tool saves that copy back. In a diagnostic outside the measured benchmark, reading through the legacy API and saving left these fields alone, while reading through v1beta1 and then saving reproduced the diff without the MCP server. The behaviour is the same in v2.2.0 and in the fixed build.

We chose not to fix it. From the v1beta1 read the tool cannot tell a user-set 0 from a server default, so stripping fields would risk the silent corruption this audit looks for. Changing the read path might avoid it; we did not build or test that. The added values are Grafana's own defaults, and we did not test whether a save from Grafana's UI writes the same fields, which would make this ordinary platform behaviour. GF-21 has not been through a refuter. It accounts for the four benchmark runs that failed.

What held up#

• Grafana's API rejects stale saves: 10 of 10 on both versions for classic dashboards, and 3 of 3 for a v2 dashboard PUT with a stale resource version.
• Patches through the tool made one after the other lost nothing: 0 of 10 pairs on both versions.
• Grafana 13's Kubernetes-style API serialised simultaneous saves: one of two PUTs carrying the same resource version was rejected in 30 of 30 pairs.
• The rule update keeps what it is given: with every optional field supplied, nothing was lost in 3 of 3 on both versions.
• A default rule update did not downgrade a rule provisioned through the API (GF-07): 0 of 3 on both versions; the call returned an error and the provenance stayed api.
• Two candidates were refuted: update_datasource sending no secrets is intended, and Grafana keeps stored secrets when they are omitted (read in the code on Grafana's main, not run live); the Agent Observability upsert requires a version field by contract.

Benchmark: what fixing the friction changes#

The fixed build made 15% fewer MCP tool calls than v2.2.0 over the same thirty runs (225 against 264) in 12% fewer turns. The reduction is concentrated in calls the usability changes target: rule reads (alerting_rules_read) fell from 63 to 38, where A1 put paused state, receiver and datasources in the list, and repeated verify reads after a write fell from 8 to 0, where B1 and B2 return the result. Attribution is by inspection, since only the combined build was measured.

This benchmark does not measure the six fixes. Each run is one agent working alone, so no two writers race, and in the alert-rule tasks the baseline agents re-sent the rule's fields, so the checker found none reset. Both builds pass the same 26 of 30 runs. The evidence for GF-01 and GF-03 is the probes above.

Method: eight tasks, two repetitions each per Grafana version, with the same model (claude-sonnet-5-5), task files and checkers for both builds, and the seeded state reset and hash-checked around every run. One task, the v2 dashboard edit, ran on Grafana 13 only, because Grafana 11.6.16 serves no v2 dashboards: fifteen task-and-version cells and thirty runs per build, not thirty-two.

Calls count MCP calls to the Grafana server; the client's own tool-search calls (36 and 34) are excluded. Per run the median was 6 [3–27] on v2.2.0 and 4 [3–21] on the fixed build. Per task, values are medians with [min–max] over four runs per build (two for the v2 edit), pooling both Grafana versions. Bold marks non-overlapping ranges, which at this size is descriptive, not a significance test.

• One task got worse: alert hygiene rose from 32 calls to 38 across its four runs. In one run the agent followed the fixed build's new description, sent partial rule updates without title, was rejected three times and re-sent every field. Those are the fixed build's three failed calls.
• The four failing runs are the same in both builds: the dashboard-edit and bulk-change runs on Grafana 13. Each made every requested panel change and failed only the criterion that dashboard-level settings stay unchanged, which is GF-21.
• Cost rose although calls fell: output tokens fell from 67,890 to 63,584 and cache-read tokens from 3.67 million to 3.42 million, but cache-creation tokens rose from 228,351 to 327,923. The records do not isolate why.

Limitations#

• Two repetitions per cell show spread, not a distribution; one model, one prompt per task, one client.
• The builds ran at different times, not interleaved: the baseline at 20:01 UTC on 9 November and the fixed build at 00:41 UTC on 10 November. Drift in model serving or prompt-cache state lands in the difference between builds.
• The six usability changes were designed from friction reported on these same tasks, so the tasks favour the fixed build by construction.
• A first attempt recorded thirty error runs because the runner did not pass the API key; each exited within seconds with no agent work. Those runs stay on the record and are not counted here.
• No new harness freeze was recorded for the fixed-build run. The 25 task, checker and agent files still match the hashes frozen before the baseline, but the script that launches the runs is not among them and was re-created; it gave each run a random nonce, where the baseline used one nonce for all thirty.
• The starting state includes leftover objects the seed does not own, present for both builds and not compared, and the agent's reading of the 80 tool descriptions is part of what is measured.

What this means for teams running the Grafana MCP server#

• Don't let two writers patch the same classic dashboard at once through update_dashboard on v2.2.0; serialize dashboard writes. If an edit goes missing, look in the dashboard's version history.
• Treat an alert-rule update as a full replace on v2.2.0: read the rule first and send every field you want to keep, including labels, annotations, paused state and notification settings.
• After a patch, confirm the end state. On v2.2.0 a patch path is an array index that points at a different panel once someone inserts or reorders panels, a misspelt final path segment is accepted, and a successful save returns a version, not the fields that changed.
• On Grafana 13, a panel-only dashboard edit can also write five dashboard-level defaults and raise the schema version (GF-21).

How findings were checked#

Each of the six fixed findings is a falsifiable claim checked against the pinned source and, where the stack allowed, a scripted probe with a control through Grafana's HTTP API, then given to a separate refuter agent in a clean context.

1. Static pass over the 131 tool registrations at the pinned commit, producing twenty candidates.
2. Probes for the seven candidates that could be tested live: 26 scripted runs, each tool run paired with a control run, with the seed state reset and its hash checked after each. The probe harness was hashed and timestamped at 18:59:37 UTC on 9 November, 39 seconds before the first run. Of the six fixes, GF-18 alone rests on code reading.
3. Refutation: a separate refuter agent in a clean context read the code at the pin, the probe results and sample captures, searched for duplicates and checked main. The refuters ran on Claude Sonnet 5.5, the same model as the static pass, so this is a second reading with fresh context, not an independent model. GF-21 and the six usability changes did not go through a refuter.
4. The two security candidates we began with — a query tool reachable under read-only mode, and a header-override path — were dropped in triage: checked against the pinned source and Grafana's own documentation, both are documented, intended behaviour, not vulnerabilities.
5. Benchmark: the task, checker and agent files were hashed, and the hash timestamped by two RFC 3161 authorities, at 20:01:40 UTC, one second before the baseline's first run. The probe and baseline logs are hash-chained, anchored and timestamped. The fixed-build log is hash-chained with a signed local checkpoint; its off-box anchor and timestamps were pending at the time of writing. The evidence is not published with this article.

What the checking caught: the simultaneous-save control was not clean, which is why the headline is a bypassed version check and not a race the tool causes. Defects in our own tooling are on the record too: the setup step was launched twice and its first harness freeze voided, and a checker bug was fixed before any agent ran.

Rebuild the fixed server#

The build used in the benchmark is public on the IsoKron fork at tag bench-r3: the unmodified v2.2.0 tag plus thirteen commits carrying the twelve changes. The branch iso/write-path has since moved on. It holds everything in the four pull requests as they now stand: the twelve changes plus eight made while preparing them for review. Those eight are saving patches through the Kubernetes-style API, a version guard on classic full-body saves, required and nested fields in partial rule updates, an error in place of an empty list when the receiver filter cannot be applied, the annotation result shape, ISO times without an offset read as UTC, quoted selector values, and the order of removals within one patch. They were not benchmarked. Each pull request carries unit tests for its changes. The probe and benchmark harnesses are not public, so this section covers rebuilding the server, not re-running our measurements. To compare the builds, build both from source, point the same MCP client at each, run the same job against a seeded Grafana, and compare tool calls, turns and failed calls. To check GF-01, release two patches to one dashboard together, then repeat with direct saves that carry the version and overwrite: false. The benchmark figures measure the combined set of changes. The build is a research experiment, not an official Grafana release; use the official server for real work.

About this research#

IsoKron performs fixed-price MCP server audits with the same method: real agent tasks against a pinned build, write-side probes with raw-API controls, clean-context refutation, upstream filing, and a before/after benchmark recorded on a hash-chained log. Details: MCP server audits.