1,000 compiler-labeled edits: measuring verification accuracy

A verification tool needs an independent reference verdict. The highest-risk error is a false green: the tool reports a broken edit as clean. A false red reports a clean edit as broken and can add an unnecessary repair turn.

The check-gate benchmark derives labels from the C# compiler, outside the Fuse verdict path. This avoids circular evaluation, where the implementation under test also defines the expected answer.

Let the compiler label the data

Fuse's check gate (fuse eval checkgate) generates its own test cases mechanically and lets the C# compiler decide the correct answer for each one.

Roslyn syntax rewriters produce two classes of single-file edit over an in-repo fixture. Breaking edits rename a bound member access, swap a used type for an undefined one, change an expression to an incompatible literal, or delete a referenced private member. Neutral edits reorder members consistently, add redundant parentheses, or insert a comment.

The compiler assigns the label. The harness keeps a breaking mutant only when compilation reports an error in the edited file. It keeps a neutral mutant only when compilation remains clean. Generation is deterministic from a recorded seed.

Now fuse_check runs against each edit and its verdict is compared to the compiler's label.

An abstention counts as neither false-green nor false-red. It records that the available compiler path could not produce a supported verdict.

The result

The recorded run (fuse eval checkgate --mutations 500) produced 1,000 compiler-verified cases, 500 breaking and 500 neutral, over the OrderingApp fixture, plus 8 curated cases: zero false-green and zero false-red verdicts (checkgate.json).

This result applies to the OrderingApp fixture, the four breaking mutation families, the three neutral mutation families, and the curated cases. It does not establish the same rate for every C# project or edit shape.

Agent-loop measurement

Edit classification does not establish task completion. The agent-loop comparison (fuse eval loop) therefore uses a separate gold-test oracle. After each rollout, the harness checks out the task's changed tests onto the agent's final edit and runs them. A rollout passes only when those tests pass. If the agent reports green and the oracle fails, the harness records a false-done.

The arena is below the full size target (15 of 24 repositories with build capture, 59 of 60 verified tasks), so the recorded run is reduced-scope. It uses one model (claude-sonnet-4-6), verified fail-to-pass tasks from the pinned library corpus, and two rollouts per arm. Of 236 planned rollouts, 234 were scored (loop.json).

• The Fuse arm recorded true pass@1 of 89 percent (95 percent CI 82 to 95), 8 false-done rollouts, and 3.1 mean agent-visible build and test calls.
• The native arm recorded true pass@1 of 82 percent (95 percent CI 74 to 90), 9 false-done rollouts, and 3.2 mean agent-visible build and test calls.

The observed pass@1 difference is 7 percentage points, but the confidence intervals overlap. The run supports reporting the observed values, not a general causal claim about all agents or repositories.

The missed gate

Three success criteria were set before the run. Two hold: pass@1 within 5 points (Fuse is 7 points higher), and false-done at most the native count (8 versus 9). The build-call criterion misses. It required the Fuse arm's build and test calls to be at most half the native count; the measured means were 3.1 and 3.2.

The harness counts speculative fuse_check turns separately from agent-visible builds and tests. On this arena, the result does not show the proposed reduction in build calls.

Reproduce it

Both suites are deterministic where they can be and gated where they cannot.

# 1,000 compiler-labeled edits (deterministic, offline):
fuse eval checkgate --mutations 500

# agent loop comparison (model-driven, opt-in, records the harness state without a model
# unless you set FUSE_LOOP_RUN=1 and have the claude CLI on PATH):
fuse eval loop

Every figure above is written to tests/benchmarks/results and reported in full, including the weaker modes, on the benchmarks page. The reproduction page documents the corpus, prerequisites, result files, and model-driven suite controls.