Benchmarks

Fuse answers a .NET coding agent's edit-time questions from a warm persistent index and, when available, a build-captured Roslyn compilation. This page reports the current recorded benchmark suites. The main question is practical: can Fuse verify an edit honestly, resolve .NET wiring, package branch context compactly, localize an open-ended task, and refuse to return junk on a low-signal query.

The yardstick is compiler-grade behavior, not tokens saved. Token efficiency shows up in scoped output: a pull request's review context arrives at a median of 1,026 tokens.

Three readers are served below. A stakeholder can read the suite headlines and the honest limits section. A new engineer can inspect the isolated harness and recorded results. A maintainer can read each suite's method, sample, index-mode distribution, and the cases where Fuse is weak.

The benchmark source sits in tests/benchmarks/Fuse.Benchmarks and the separate Fuse.Benchmarks.slnx solution. Fuse 4.4 does not package an evaluation command. Each archived scorecard lives under tests/benchmarks/results/<suite>.json.

An honest caveat on index mode

Fuse's semantic graph is strongest when a repository loads with a build capture: a full MSBuild semantic workspace restored from a real dotnet restore (a project.assets.json on disk). Without that, indexing falls back to syntax mode and semantic suites read a thinner graph.

The harness restores each checkout before indexing with the --restore flag (see Reproduce the Benchmark). It records the achieved index mode per repository, so each number can be read against how much structure was available. Where a repository restores cleanly, it loads semantically. Suite A is the semantic ceiling. The corpus suites run mostly in partial or syntax mode on some repositories, so read them against that ceiling.

The corpus

The current corpus manifest (corpus-v2.json) pins 24 open-source .NET library repositories by commit. Corpus health records 15 of 24 at the build-captured semantic tier. The review, localization, and ranking scorecards select 69 merged changes across 23 repositories, three per repository; AutoMapper had no qualifying reconstructed change. PR ground truth (prs-v2.json) keeps a merged commit when it changes 2 to 25 C# files and its title describes a code change. Maintenance commits (CI changes, dependency bumps, reverts, formatting passes) are excluded because a title alone cannot locate their change set.

Historical corpus (superseded, for provenance only)

An earlier mixed corpus (three libraries plus the eShopOnWeb ASP.NET Core application, prs.json, 53 PRs) is preserved in *-retired.json result files. It is provenance, not the primary corpus. Application external validity now has its own supplementary eShopOnWeb scorecard described under Suite B.

Reproduce

Prerequisites: the .NET SDK 10.0 or later and Git. Corpus-bound suites skip gracefully when the pinned corpus is absent, so a bare invocation stays offline. The semantics suite runs on the in-repo fixture with no corpus.

The source remains available in Fuse.Benchmarks.slnx, but it is excluded from the product build and the packaged CLI. A repository that could not restore on the recording machine remains in syntax mode in the archived scorecard. Each scorecard includes per-task hits, misses, and the index-mode distribution. Model-driven suites are not byte-reproducible.

The isolated harness and recorded-run leaderboard are in Benchmark source and recorded results.

Suite A: semantic resolution

Suite A checks the extracted semantic graph against a hand-built edge ground truth on the in-repo OrderingApp wiring fixture. The fixture covers .NET wiring kinds that a regex or tree-sitter scan cannot follow: DI registration and constructor injection, MediatR request-to-handler dispatch, ASP.NET route-to-action mapping, options binding and consumption, hosted-service workers, MediatR pipeline behaviors, EF Core DbContext-to-entity and entity-to-configuration wiring, Scrutor decoration, factory-lambda registration, minimal-API endpoints, gRPC services, and SignalR hubs. It also includes precision edge cases: an explicit open generic registration, a TryAdd registration, and a multiple-implementation ambiguity where only the registered implementation resolves.

On the OrderingApp fixture, Fuse matches all 24 adjudicated wiring edges. This result does not establish 100 percent wiring accuracy for arbitrary repositories.

A corpus sampled-adjudication mode in the separate semantics suite indexes the corpus repositories, extracts predicted graph edges, samples a fixed number per edge type with a seeded shuffle, and writes them to semantics-corpus-sample.json for adjudication. A repository that loads syntax emits no graph edges to sample. One nuance: the constructor-injection edge captures any constructor-parameter dependency, including value and enum parameters, not only DI-container services.

Suite B: change impact and review

Suite B measures how compactly fuse review packages branch context after Git supplies the changed-file set. The primary run covers 69 real merged changes across 23 of the 24 corpus repositories at a 25,000-token budget with --restore on. Changed files are seeded as must-keep, then Fuse adds semantic blast-radius context. This suite measures packing precision, not discovery of the files a change needs.

Index modes over the 69 PRs: semantic 33, partial 18, syntax 18. A majority of PRs now load with real semantics (versus the historical corpus, where only 1 of 53 did), so the semantic blast radius is exercised on real code. Versus a grep baseline (rank C# files by title-token matches, admit to the budget), review reaches 67 percent changed-file recall at 8 percent precision.

How to read the precision:

• Changed-file recall is 100 percent by construction, since changed files are seeded as must-keep. The signal is precision (93.4 percent) and F1 (0.966).
• Review delivers a PR's context compactly. The median PR is scoped in 1,026 returned tokens.
• The ground truth is the reconstructed changed-file set. On 54 of the 69 PRs the change touches the public or protected surface.

The historical corpus (review-retired.json, 53 PRs) recorded 100 percent recall at 79.8 percent precision with only 1 of 53 PRs loading semantically.

Application external validity

Because the primary corpus is library-focused, a supplementary run records review on eShopOnWeb (ASP.NET Core: DI, MVC and Razor Pages, EF Core, config sprawl), pinned at the same commit as the historical corpus (review-app.json, corpus-app.json, 8 reconstructed PRs). Build capture now loads eShopOnWeb's main checkout at semantic mode. Review over its PR worktrees (partial 6, semantic 2) scores 100 percent changed-file recall at 84.2 percent precision, median 366 returned tokens, beating a grep baseline of 73 percent recall at 10 percent precision. Precision reads a little below the library corpus's 93.4 percent because an application's blast radius legitimately pulls in wiring and support files a reviewer wants.

Suite C: open-ended localization

Suite C is the open-ended floor. It gives fuse localize only a PR title, with no Git base, and asks it to find the changed files from the title text alone. This is the hardest mode and the weakest measured result. Retrieval uses the lexical channel (keyword search over the index). The mechanism (subword identifier indexing, stemming, a comment bridge, and structural priors) is described in Retrieval Internals.

On the primary corpus (69 PRs, title only, restore on):

By task category: identifier-rich titles 52 percent recall, natural-language domain titles 24 percent, test-only titles 46 percent. Localize main-checkout index modes: semantic 14, partial 4, syntax 5. Open-ended recall on the primary corpus is 37.7 percent versus 15.0 percent on the historical corpus (localize-retired.json). The reconstructed set drops maintenance titles, so it carries no no-signal tasks here. Low-signal refuse-and-route behavior is measured on the historical corpus, which did carry them: correct refusal was 100 percent at 0.0 percent false rejection.

No embedding channel

The shipping retrieval path is lexical and ships no embedding model. The retired mixed-corpus results preserve earlier embedding experiments for provenance, but embeddings are absent from the current product and from the current ranking configurations.

The signal-sufficiency contract is why the precision metrics hold despite the low recall. Every request is graded confident (a candidate stands clear; return the tight set), partial (some signal, no clear winner; return a small flagged set plus a navigation map), or insufficient (no usable anchor; return only a navigation map and ask for a symbol, route, service, request, config section, or git base). The grade is computed from the candidate score distribution with no model. Correct refusal on no-signal titles was 100 percent on the historical corpus (low-signal detection F1 1.0), with false rejection on answerable queries at or under the 0.10 target. With a Git base available, the routed mode is change impact (Suite B). Suite C reports what Fuse can do from a sentence with no base.

Ranking regression (the ranking gate)

A field-weight inversion once shipped on the localize hot path (the FTS5 path column was weighted highest, so a folder-name match outranked a symbol-name match) because nothing measured ranking. The separate ranking suite closes that gap. It scores retrieval ranking with MRR (mean reciprocal rank of the first correct file), recall@k (fraction of tasks where a correct file appears in the top k), and nDCG@k (ranking quality in the top k) against changed-file ground truth. The recorded comparison includes lexical-only, the shipping lexical-plus-centrality configuration, and the now-removed git co-change prior. It writes results/ranking.json and is the required gate on any change to field weights, tokenization, query expansion, or priors.

On the primary corpus (69 PRs, index modes semantic 14 / partial 4 / syntax 5):

Both are materially higher than on the historical corpus (ranking-retired.json, MRR 0.197, recall@10 15.0 percent). On the primary corpus the git co-change prior is net-negative: it costs MRR -0.055 and recall@10 -1.9 percent. Fuse 4.4 removes the collector, stored data, and runtime configuration; the archived comparison remains as evidence for that decision.

Peer comparison (Fuse versus CodeGraph, coa-codesearch, and Serena)

The peer layer gives each tool the PR title and scores the returned file set against the changed files. Fuse runs through its index-then-localize path. CodeGraph (offline tree-sitter graph) runs codegraph init then codegraph explore. coa-codesearch (a .NET Lucene MCP server) and Serena (an LSP-backed symbol-search MCP server) are each driven by one headless claude rollout restricted to that server's search tools. At a 50,000-token budget (layer6-peers.json):

The deterministic arms (fuse, codegraph) ran all 12 sampled PRs. The model-driven arms (coa, serena) ran one claude rollout per PR, bounded to one PR per repository (4 PRs total). On the deterministic 12-PR comparison, Fuse leads CodeGraph on recall (19 versus 9 percent) at higher precision. On the four shared model-driven PRs, Serena's higher aggregate is dominated by a single tiny-repo outlier (on a 2-file change in a 45-file repository its symbol search returns the whole repository and scores 100 percent recall). On the two substantive PRs Fuse leads or ties. coa returns a long ranked path list, so recall is moderate but precision is near zero. Token columns are not directly comparable: Fuse and CodeGraph return source; coa and Serena return path or snippet lists.

Reproduce the head-to-head:

# Fuse and CodeGraph only (deterministic, all sampled PRs)
pwsh -File tests/benchmarks/harness/layer6-peers.ps1

# Add coa-codesearch and Serena (model-driven, one claude rollout per PR, bounded per repo)
COA_CODESEARCH_EXE=/path/to/COA.CodeSearch.McpServer.exe \
SERENA_CMD=uvx \
pwsh -File tests/benchmarks/harness/layer6-peers.ps1 -PerRepo 4 -ModelPerRepo 1

The run writes tests/benchmarks/results/layer6-peers.json. A full 50-to-100-PR model-driven run needs a larger compute budget.

Suite E: token reduction and fidelity

Suite E counts, for each repository, the raw o200k_base tokens of the C# files, reduces them at each level through the shipped reduction path, counts the reduced tokens, and at skeleton level checks how many public and protected types and methods survive. Fidelity is counted by parsing the raw source with Roslyn as independent ground truth. The separate reduction suite produces the archived result.

Percentages are token reduction against the raw C# token count. Across these four repositories, standard removes 11 to 41 percent, aggressive removes 16 to 46 percent, skeleton removes 38 to 44 percent, and public API removes 47 to 60 percent. The fidelity check compares public and protected declaration names against an independent Roslyn parse: skeleton retains 100 percent of type names and 96.3 to 99.4 percent of method names in the recorded repositories. It does not claim complete API preservation.

Suite F: the fuse_check honesty gate

fuse_check is only useful if it does not lie. A false green reports a broken edit clean (dangerous: an agent could commit a change that does not build). A false red reports a valid edit broken (wastes an agent turn). Suite F measures both.

It builds a small self-contained compilation with raw Roslyn (no MSBuild), then applies single-file edits each with a known-correct verdict: equivalent rewrites, an added valid overload, and a comment-only change that must stay clean; a call to a missing member, a wrong return type, an undefined type, a syntax error, and an assignment to an init-only property that must each be flagged. An abstention counts as neither false green nor false red. The separate checkgate suite writes results/checkgate.json. The eight curated edits are all classified correctly: zero false green and zero false red.

Mutation-derived honesty at scale

Eight hand-built cases are too few to carry the "never lies" claim, so the separate checkgate suite generates ground truth mechanically. Roslyn syntax rewriters produce breaking and neutral single-file edits over an in-repo fixture. The verdict is confirmed by the compiler: a breaking mutant is kept only when the compilation reports an error in the edited file; a neutral mutant only when the compilation stays clean. Generation is deterministic from a recorded seed.

The recorded run (--mutations 500) produced 1,000 compiler-verified cases (500 breaking, 500 neutral) over the OrderingApp fixture: zero false green and zero false red (0.00 percent over 1,000 verified cases). The SampleShop fixture (a two-project web solution) does not bind in the suite's flat in-process compilation (CS0234/CS0246/CS0103 without full per-project references), so it is recorded skipped. The gate is false green 0 and false red under 1 percent; the recorded run meets it.

Fast check versus full build agreement

fuse_check answers at the best available verification grade: a speculative typecheck against a build-captured compilation (oracle grade), or, when no build capture is available, a real dotnet build scoped to the owning project (build grade). The recorded sub-second timing applies to the opt-in resident workspace. Build grade is ground truth by construction. The honesty question is whether the oracle path agrees with it.

The separate checkgate suite runs the same proposed content through both paths for a sample of mutants with a build-capture worker provisioned. The recorded run sampled 24 OrderingApp mutants: all 24 comparable with no abstention on either side, diagnostic-id agreement 24 of 24 (100.0 percent), clean/red verdict agreement 24 of 24. The gate is at least 99 percent agreement, met at 100 percent on this sample.

Loop metric (model-dependent)

Per-payload token reduction does not establish lower cumulative session cost. The loop suite measures how many build-gated turns an agent takes to finish a coding task correctly.

pass@1 means the agent's edit passes the project's own tests on the first attempt. The harness checks this with a gold-test oracle: the task's changed tests are checked out onto the agent's finished edit and run, so success means the tests pass, not that the transcript claimed green.

One Claude Code CLI driver resolves each task (edit, verify, repeat) in two arms:

• native: filesystem tools plus dotnet build and dotnet test
• fuse: Fuse MCP tools, where verify can be a speculative fuse_check instead of a dotnet build round-trip

The run records per arm whether it reached green, iterations to green, agent-visible dotnet build and dotnet test round-trips (counted apart from speculative fuse_check turns), and false-done (the agent declared green but the oracle disagreed).

The deterministic core (turn classification from stream-json transcripts and metric computation) is unit-tested and runs offline. Model-driven arms are opt-in in the separate loop harness. FUSE_LOOP_RUN=1 with the claude CLI present launches rollouts. The harness writes results/loop.json.

Recorded run (loop.json)

Recorded on the primary corpus arena: 59 verified oracle tasks in corpus-tasks-v2.json, where each task's changed tests fail at the base commit and pass at the merge commit. Driver: claude CLI version 2.1.181, model claude-sonnet-4-6. Two rollouts per arm per task; 236 planned, 2 wedged and omitted, 234 scored (118 fuse, 116 native). The arena is below the full size target (15 of 24 repositories with build capture, 59 of 60 verified tasks), so this run is reported with confidence intervals.

The oracle scores only rollouts whose gold tests compiled and executed over the agent's edit (84 of 118 fuse, 80 of 116 native). A rollout whose gold tests could not build against the edit is unscored, applied identically to both arms.

On this arena Fuse's pass@1 is 7 points higher (89 versus 82 percent). False-done is lower (8 versus 9). Agent-visible build-plus-test round-trips are essentially equal (fuse 3.1 versus native 3.2). Fuse's edge here is a higher test-verified success rate and fewer silent wrong answers.

The earlier single-rollout pilot (loop-pilot.json) is superseded by this run.

Short elapsed-time pilot (loop-efficiency-pilot.json)

A separate five-task pilot measured the full Claude Code session time in three arms: native tools, standard Fuse, and Fuse with the resident workspace enabled. It ran one rollout per arm on five Ardalis.GuardClauses tasks. Fourteen of 15 planned rollouts completed; one resident rollout was omitted.

The final test check executed in both compared arms for only two tasks. On those two pairs, standard Fuse recorded a median time saving of 9.4 percent and resident Fuse recorded 20.0 percent. This is a directional pilot, not a developer-productivity result: two verified pairs cannot support a public "developers are X percent faster" claim.

The broader timing rows reinforce that limit. Across all five native-to-standard-Fuse pairs, mean elapsed time was equal and median Fuse time was 30.4 seconds slower. Across the four native-to-resident pairs, resident Fuse was 68.9 seconds faster at the median. Every final test check that executed passed, but it executed for only three native, two Fuse, and two resident rollouts. The result file retains every duration, omission, and denominator.

DiagBench: repair-packet auto-apply rate

A fuse_check repair packet is only useful if its suggested fix repairs the error. The separate DiagBench harness measures that without a model: for each API-shape mutant (a misspelled member or type reference), it builds the shipped repair packet, auto-applies the machine-applicable TopRepair, recompiles, and records whether the error went to zero with no new error.

The recorded run (results/diagbench.json) is 20 single-token near-miss mutants (14 CS1061 missing-member, 6 CS0246 unknown-type). Every one carried a TopRepair and every one was auto-fixed (100 percent per class). These are one-edit typos whose original is the nearest real name. This is a benchmark-harness path only; the product never auto-applies a repair without the agent asking.

What is not measured at scale

These are honest gaps. Do not read an absence here as a result:

• The model-driven peer comparison at full scale. All four arms run head-to-head, but the model-driven arms run a bounded 4-PR sample. A full 50-to-100-PR model-driven run needs a larger compute budget.
• Elapsed task resolution at scale: the five-task timing pilot has only two verified pairs. A publishable time-savings estimate needs a larger task sample with final tests executing in both compared arms.
• Cross-machine timing. The warm-index latency model is on Performance and Benchmarking; no fixed cross-machine number is published.

How to read these results together

Suite A is fixture-scoped: Fuse matches 24 of 24 adjudicated wiring edges in OrderingApp. Suite B shows that Git-seeded branch context is compact: 93.4 percent precision and a median 1,026 returned tokens, with changed-file recall fixed at 100 percent by must-keep seeding. Suite C is the open-ended floor: from a title alone with no base, recall is 37.7 percent. The loop suite records a higher test-verified success rate, while agent-visible build-plus-test counts remain essentially equal at 3.1 for Fuse and 3.2 for native. The primary corpus is library-focused; the supplementary eShopOnWeb run records 100 percent Git-seeded changed-file recall and 84.2 percent precision.

Related pages

The warm-index model and how to think about Fuse's speed are in Performance and Benchmarking. The retrieval mechanism is in Retrieval Internals. The scoping modes are in Scoping To What Matters. The harness, corpus manifest, PR ground truth, and suites live in tests/benchmarks and its readme. Full reproduction steps are in Reproduce the Benchmark.