Open Forward Reviews

The short version

Task by task

Eleven tasks, each run three times with the tooling and three times without, on the same model and the same starting repository. "Better" and "cheaper" below always mean the equipped agent compared with the same agent given nothing.

What that adds up to

The outcome column barely moves. Across the eleven tasks, two improved, one came out worse, and eight showed no change — and every one of those is a count of three against three, which the calibration further down says sits inside the judge's own margin of error. Read the column as "nothing separated", not as a scoreboard.

The cost column moves a great deal, and it moves both ways. Where a skill is actually loaded, the direction is a property of the task:

• a review — same result, and every equipped run cheaper than every unaided run, with input tokens roughly halved. The tooling paying for itself.
• an upgrade — the widest gap on this page, the other way round: ten to seventy times the input tokens, one run at $1.23, and neither side finishes the job.
• documentation — about twice the spend, and the largest outcome move on the page: 0 of 3 to 2 of 3.

Where no skill was loaded, spend is mostly flat — which is what nothing being loaded predicts. So the summary is not that the tooling makes no measurable difference. It is that its clearest measured effect is on spend, that effect is large, and its sign depends on the task. Whether the work comes out better is the question three runs per side cannot answer, and the sections below say so one task at a time.

Two bugs from the world, and what they cost

Every case above was written here. The two below were not: the request is a user's bug report, quoted, and the answer is the one the maintainers' own fix established — a test that fails at the commit before the fix and passes at it, verified both ways before either case was built. Nobody in this project chose what counts as correct.

Three runs of twenty-four fixed a real reported defect, and the tooling was never opened on either task — the requests describe an artefact, and no skill's first sentence names that work. These two tasks are also the most expensive here by an order of magnitude: half a dollar and millions of input tokens per run, against a cent or two on the tasks written for this benchmark.

Does making a repository agent-ready help?

Everything above varies what the agent carries. This varies what the repository carries, which is the question the stack is actually about — and until now it had never been measured here. Same defect, same model, same fleet; the only difference is eighteen agent-facing files generated into the repository by agent-rules: a root AGENTS.md, five scoped ones, and their symlinks.

The difference from every other comparison on this page is that the treatment arrived. Five of the six runs read the generated AGENTS.md, mentioning it between 3 and 43 times. Skills, by contrast, were loaded in 0 of 6 runs on this same task — so those comparisons measured the base model twice, and this one did not.

What is being measured

Whether an agent equipped with the Netresearch engineering stack takes a realistic, deliberately underspecified request and works out for itself what the job is — and what that equipment costs when it goes unused.

Every case starts from what a developer actually has: the extension repository checked out and a TYPO3 instance running with that extension installed, served over HTTP, reachable under the hostname the project's own DDEV configuration declares. Not a bare checkout. A tool that reads a running application has nothing to read in one, and a benchmark built on one measures how much a tool helps someone working in an environment nobody has.

Case 1 — a review: same answers, half the cost

Four configurations, three runs each, no errored trials. Quality is flat — every configuration produces a substantive review and nothing separates. The cost does separate, and cleanly.

The ranges do not overlap. The dearest Netresearch run costs less than the cheapest unaided one — $1.55 against $2.21. Two groups of three separate completely by chance one time in ten, in one direction or the other: enough to act on, not enough to call established, which is why the comparison deepens where it sees one. This page said one in twenty until 20 August 2026 — that is the figure for a direction named before the data were seen, and this one was not.

The saving is not in doing less. Tool call counts overlap; input tokens do not. The agent takes a similar number of steps and carries roughly half the context through them. In all three runs it reached exactly one skill of the nine — the conformance skill — and never opened the rest. A procedure that has been adopted is one that does not have to be reconstructed.

Case 2 — a runtime bug: nothing offered was taken up

Five configurations against an installed TYPO3 13.4 whose database is in the state a real production report described. Every configuration solves it. Nothing separates — not quality, not cost, where the unaided agent alone spans $4.20 to $10.46 on identical inputs.

Across all fifteen runs, not one skill was invoked — with nine available, with twelve, with thirteen. Both MCP configurations did reach their servers, so the capabilities were demonstrably present. The skills simply went untouched: no procedure on offer fits a runtime diagnosis, and the agent boots the framework itself instead.

Case 3 — an upgrade: the first result a judge did not decide

This case has mechanical ground truth. The verifier takes the tree the agent left, pins the dependency matrix to each TYPO3 line, resolves it and runs the extension's own test suite. An arm either produced something that installs and passes on both lines, or it did not.

Both Netresearch configurations completed the upgrade in every run; the unaided agent in two of three; the companion in none, one of its trees failing to resolve at all. Three runs make that a reason to spend more, not a finished claim — and the deepening stage that was spending them died on the subscription's rate limit.

Cost separates nothing here and is an order of magnitude above the other cases. The companion's cheapest run, at $1.23, is the one that produced the unresolvable tree: cheap because it stopped, which is why a cost column is only readable beside an outcome column.

Case 4 — a contract eval: does the check fire

A deliberately different kind of case, and the cheap one. Four Extbase properties declared private in a minimal extension, two protected ones beside them, the expected answer known. No judge, no instance, no network beyond the model endpoint — it runs in seconds and can gate a merge, which an open review cannot.

Graded in both directions: the four must be named and the two must not be called broken. Control scores 0.97, the Netresearch configuration 1.00.

What the four Opus cases say together

The same configuration is the cheapest arm on one case and the dearest on another. nr-full runs the review for $1.34 and the runtime diagnosis for $7.43, against an unaided $2.80 and $5.03. What a stack is worth is a property of the task it meets, and a benchmark that reports one number for it is answering a question nobody asked.

A second model, and every case at once

On 21 and 22 August 2026 the whole benchmark ran on claude-haiku-4-5-20251001: nine cases, randomised block series, three trials per arm, 131 graded runs in all. Every trial passed the validity gate; none was discarded. The four sections above stay as they are — they are Opus figures, and putting the two models in one table would place numbers side by side under a heading that names neither.

The declared endpoint moved in none of the nine. That is the headline and it is not the interesting part, because three trials per arm cannot establish a difference and the stopping rule says so before the run starts. What the sweep does show is a pattern that no single case could.

Whether a skill is reached for at all

Counted over every equipped trial on record rather than over one series — every job on disk whose fleet resolves to at least one skill, across both models and every ablation, 131 graded runs and the earlier ones besides. The counts are computed by scripts/invocation-census at build time, because the hand-written version of this table was stale within a week.

The table above is a real split, and the column in the middle is the old reading of it — that requests naming a domain route and requests describing a task do not. Four experiments on 28 August replaced that reading, and one row already contradicts it: the restraint case is phrased as a task and now routes 6 runs in 6. What actually predicts the right-hand column is stated below the next two paragraphs, which are kept because each is still true of the case it describes.

Sometimes the capability is simply not there. The organisation publishes 42 skill repositories; the fleet under test carries eight, and nr-full twelve. Neither holds a release, git or forge skill. The release case asks to "prepare the 2.4.2 release", and netresearch/github-release-skill — which exists, and whose description activates on the word "release" — was not on offer. That case therefore measured fleet composition, not routing, and an earlier version of this page said otherwise.

That one has now been settled by adding the skill. Twelve trials, the same fleet plus github-release-skill and nothing else: skill invocation went from 0 of 6 to 6 of 6, Fisher exact p 0.002 — the first declared result on this page to separate at a p a conventional threshold accepts. Routing was never the constraint. It is also why the release row above reads 6 of 16 rather than 0 of 16: the two arms carried different fleets, and the sub-line under the count says which. Loading the skill did not make the work come out right, though — the mechanical check stayed at 0 of 6 on both sides. The record.

Sometimes it is there and described for a different job. The same release case had typo3-docs in the fleet, shipping scripts/check-guides-xml-version-sync.sh — a script whose only purpose is the check every trial failed. Its description names guides.xml and never mentions releases or version bumps. That is a routing observation about one skill, and unlike the first it is testable in one series: change the description, re-run the case.

That series has now run twice, and the answer is where the words sit. Both rounds are twelve trials on the restraint case, 28 August, identical in everything but one skill: typo3-conformance at its release against the same skill on a branch naming the artefacts the case is about — version declarations that disagree, composer.json against ext_emconf.php, supported TYPO3 versions. Nothing was removed either time. Skill invocation was declared as the endpoint before the first trial of each.

A description is not a bag of keywords. The same words, moved earlier, take this task from never routing to always routing — and unlike the first attempt the rewrite fits the skill repository's limits, so it ships. The full record, including what did not follow: the outcome is 3 of 6 on both arms.

What predicts whether a skill is reached

Six measurements on 28 August, each with the endpoint declared before the first trial. An earlier version of this page read one rule out of them — a skill is reached when the words the request itself uses appear in the opening clause of its description — and that rule is wrong. Checking every silent case against the opening clause of every skill in its fleet, five of six share vocabulary and none of them route: a skill describing TYPO3 upgrades shares "extensions, typo3, versions" with the metadata request, and the Go request literally opens "Someone from the security side says…" beside a security skill that was in the fleet and never loaded.

What survives is narrower. Two interventions moved routing completely, and both put the action the request asks to perform into the opening clause, in the request's own terms — not merely the same nouns. And one of those two is close to circular: that clause was written from the request, so its matching is not a discovery. The release case is the clean one, an already published skill whose first clause names the request's verb.

"No difference" was hiding two opposite effects

The release check is a gate over the four places a TYPO3 extension states its version, and it reports one line. Both arms fail it, so the aggregate says nothing happened. The four places separately say something else.

The unaided agent is consistent: two places every run, the other two never. The equipped agent splits in half. Three of its six runs went further than the unaided ones — 27 to 36 steps against 19 to 28 — and updated guides.xml, which no unaided run ever touched. The other three never found the extension at all: they searched the running TYPO3 instance the agent starts in, and closed after eight to ten steps by asking which repository was meant. No unaided run did that; all six found the checkout.

Where the skill did fire, the direction is not the same

Three cases, three different trades. On the review the equipped arm reaches the same result for about a third less — every equipped trial cheaper than every control trial. On the upgrade it spends ten to seventy times the tokens and neither arm finishes; what it buys there is verification and prioritisation of an upgrade nobody completed. On documentation it costs roughly twice as much and is the only case in the sweep where the equipped arm moves a dimension at all.

Read the upgrade's control column before drawing a conclusion from its cost: four to five tool calls on a TYPO3 major-version upgrade, about 100k input tokens, two to three cents. The unaided agent barely engages, and a cheap non-attempt is not a cheap answer. The same shape appears in the restraint case, where two control trials answered with zero tool calls — correctly, since nothing needed changing — and scored the two lowest results in that series. An empty diff is what both a careful check and a non-attempt produce, and only a rubric that asks what was established can tell them apart.

All of these cost lines are exploratory. Each series declared one endpoint before its first trial, and none of them was cost. A completely separated secondary line names the next experiment; it is not the result of this one.

Outside TYPO3: the same model, two other languages

Two cases added on 26 August, one Go library with a reported timing side channel and one Python scripts repository whose scheduled job reports its designed hand-over as a failure. Each ran as a block series against a domain fleet — nr-go, nr-general — rather than against nr, because a case outside TYPO3 run against nr measures composition and nothing else.

Go: six trials of six closed the side channel — the regression check that fails on the parent passes on every tree — and none of the six left a test, and none used the escaping helper the neighbouring code path uses. The Go skill was on offer and invoked in no trial. Python: nobody got all three answers right; four of six turned every give-up green, the secondary limit included, and two of six changed nothing. No Python skill exists to offer, so that arm's zero is composition by construction.

The Python series also found instrument failure 23: a judge criterion that says "read the diff" was reading a transcript that never contained one, and scored a diff that makes the secondary limit green as having kept it red — twice. The mechanical check, which reads the tree, caught all four.

Running it against your own agent

This page is a report and carries no commands. Everything below is what a reader — a person or an agent — needs to produce a comparable result. Nothing here is specific to Claude Code except the default; --model and --agent are what the harness varies.

What it needs

• the repository, and Docker running — each trial builds an environment image and starts containers, and some cases start a database or a directory server beside the agent
• a credential in the environment: CLAUDE_CODE_OAUTH_TOKEN (a Claude subscription) or ANTHROPIC_API_KEY (billed per token). The agent under test and the judge both use it
• time and money: a trial takes four minutes to over an hour depending on the case, and costs between two cents and a euro on a small model

One trial, to see the pipeline work

./scripts/run-smoke OFR-TYPO3-METADATA-001 --fleet nr --model claude-haiku-4-5-20251001

That is a smoke run and not evidence: one sample of a stochastic process says nothing about behaviour, and the script's own header says so. It answers whether the environment builds and the verifier grades.

A comparison, which is what produces a result

./scripts/run-comparison OFR-TYPO3-METADATA-001 \
    --arms control,nr \
    --primary consistency \
    --model claude-haiku-4-5-20251001 \
    --seed 11

./scripts/analyze experiments/OFR-TYPO3-METADATA-001-<stamp>.json

--primary is required and is written into the experiment record before the first trial. That is the whole discipline in one flag: a run reads every dimension its case grades, and a threshold meant for one look is a coin flip at eight. The runner schedules one trial of each arm per block with the order shuffled, and stops after three trials per arm unless the declared endpoint separates completely.

The cases, and what each declares

Arms are named by the files in fleets/: control provisions nothing and is the baseline every result rests on, nr is the TYPO3 and PHP domain skills, nr-full adds the general tooling, nr-go and nr-general serve the cases outside TYPO3. An ablation states its difference from a parent rather than copying it — derives_from: nr with without: to drop a skill or at: to move one to another version.

Another agent, not only another model

--model changes the model inside the same harness. Whether a result is about the stack or about the stack inside Claude Code is a different question, and scripts/sentinel exists to answer it by holding everything constant except the harness. It has not been run. Until it has, read every figure here as scoped to one harness.

If you produce a result

Three things make it comparable with these. Declare the endpoint before the first trial. Report counts rather than means — 2/3, not 0.667, which implies a precision three trials do not have. And read the section above this one before quoting any count of three.

What a count of three can carry

The judges were calibrated on 26 and 28 August: four recorded runs, 5 gradings of each on identical input, all eight dimensions. The spread numbers were reassuring — no mechanical criterion moved at all, no dimension moved more than a third of its scale. The spread was the wrong thing to read.

Every comparison here counts dimensions met, at 0.75. Read that way, the same measurements say 8 of 32 dimension verdicts flip between met and not-met on identical input — and the same ones flip in both calibrations. They are the dimensions scoring near the boundary, where one criterion moving one step of its three-point scale crosses the line. [0.75, 0.75, 0.75, 0.67, 0.67] is a spread of 0.08 and a verdict that changes three times.

How far that reaches, measured across every experiment on record: 49 of 61 dimension rows contain at least one boundary trial, and 155 of the 282 trials behind those rows sit within one judge step of the threshold. Half the rows have three or more of six there; six rows have all six.

What is not yet measured is whether the judges are right. The bands they are checked against were written by the same person who wrote the rubric, so the calibration can establish that a judge is unrepeatable and cannot yet establish that it is correct. That gate is reported and does not fail the build until an independent rater has set the bands.

Method

Execution, isolation, skill injection, trajectory recording and regrading are provided by Harbor. This project contributes the review methodology, the cases, the rubric and the conventions that keep results comparable. Grading rests on observable behaviour only — tool calls, commands, files opened, output seen, the resulting diff and the final answer. Stated intentions are not evidence, and private reasoning is never read.

Two arms may be compared only when a fingerprint over the whole configuration matches and the provision differs: the task digest, which covers instruction, rubric, environment and network policy, plus agent, model, judge and trial count. What each arm was given is probed before it is asked what it used, because a server that never answered and a server that answered and went unused are different findings, and only one of them voids a run.

Every failure this benchmark made about itself is written down, including the ones that produced plausible numbers rather than errors — a leaked answer, a judge that scored zeros it never computed, a floor under total failure, a probe that reported "nothing provisioned" for a configuration carrying thirteen skills, an aggregation that zeroed what its own docstring promised not to zero: docs/instrument-failures.md

An external code review in August 2026 found five more, none of which any number could have revealed: three sat in files whose comments described the opposite of what the code did. Its four blocking items are now closed — a validity state per trial so an infrastructure failure can never be read as a score, exact small-sample statistics with a primary endpoint declared before the run, randomisation at the level of the trial rather than the batch, and case expectations moved out of the public repository. Twelve further items remain open.

These figures were produced by benchmark 0.6.2, 0.7.0, 0.8.0, 2.0.0, unversioned — read from the recorded jobs rather than from the repository's current VERSION, which says what the tree is today and not what produced a number. A result names the version that produced it: a case added or retired makes earlier numbers measurements of a different benchmark, and a rubric change makes them comparable only after a regrade (docs/governance.md).

Specification, cases, rubric and raw results: github.com/netresearch/agent-system-evals