enola/Benchmarks

How Enola is measured

Enola, measured on public code.

Five questions decide whether Enola's graph and its verdicts can be believed. Each is measured on public repositories, with one binary and scripts you can re-run, and where a result is unflattering it is still published.

Every benchmark run updates the full results, tables and method in docs/BENCHMARKS.md. This page shows the headline of each.

Reproducibility
95/95
public repositories produced byte-identical graphs across a cold run and two warm ones
Change isolation
20/20
injected cycles caught, with none of 1,859 pre-existing findings repeated as new
Cross-repo linking
19/24
fixture calls resolved, with the 3 unresolved and 2 external reported, not hidden
Agent loop
3/3 → 0/3
agents shipped the cycle without Enola versus with hooks, over nine trials

1 of 5

Does the same commit produce the same graph?

Every repository is indexed three times, once cold and twice warm, and the snapshot ID and the hash of the fact file are compared across all three. Running cold and warm is the point: a cached run and a from-scratch run must agree.

The snapshot ID is a hash of the facts, the version and the config, so two runs on the same commit are provably the same graph. A diff between snapshots means something only because of this.

Full, current results →
95/95
public repositories produced byte-identical graphs across a cold run and two warm ones

2 of 5

When something regresses, is exactly that reported?

Pin a baseline on a real repository and move it through four states. Only the injected cycle may fail, and it must be reported as exactly one regression, however many findings the repository already carried. The runs enforce --fail-on=cycles: Enola fails nothing unless a policy names it.

Full, current results →
20/20
injected cycles caught, with none of 1,859 pre-existing findings repeated as new
1. No change
PASS · +0 facts
2. Benign addition
PASS · 2 or 3 facts named
3. Injected cycle
FAIL · 1 regression
4. Reverted
PASS · +0 facts

3 of 5

Of the cross-repo calls that exist, how many are linked, and are the misses shown?

Committed fixtures pair services across languages (TypeScript, Go, Python, PHP, Dart, Protobuf, OpenAPI), each with a known answer. Enola must link each call to the code that serves it and report every call it could not link, instead of hiding it or guessing.

The benchmark also runs real public clusters, including ones where Enola resolves little and says so: a service whose calls it could not follow is classified as a coverage gap, never as isolated.

Full, current results →
19/24
fixture calls resolved, with the 3 unresolved and 2 external reported, not hidden

4 of 5

Does it finish on a real repository?

Every sweep indexes the whole public corpus, from small applications to the Linux kernel, GitLab and dotnet/runtime, recording time and peak memory per repository. A graph the other questions rely on has to be buildable on real code first.

Speed is recorded, not claimed: Enola is not benchmarked on speed against other tools.

Full, current results →
0
parse errors across the whole corpus, the Linux kernel included

5 of 5

Does an agent, given the check, avoid shipping the regression?

One coding task, given to Claude Code three ways: bare, with Enola's MCP server and instruction, and with the hooks that check the change when the agent stops. The obvious implementation closes a dependency cycle that compiles and passes every test.

Nine trials is a demonstration that the mechanism works end to end, not a measured effect size. No arm asked for the comparison itself; the hooks ran it.

Full, current results →
3/3 → 0/3
agents shipped the cycle without Enola versus with hooks, over nine trials
"The regression was src/domain ↔ src/store: my first attempt made the domain import { listOrders } from the store, which closed a loop with the store's pre-existing import type { Order } from the domain. I removed the offending edge."Unprompted, from a transcript with the hooks installed

What this does not measure

The full list on GitHub →

The three benchmarks that did not work.

Every page like this one shows you the measurements that survived. Here are the ones that did not, and why.

Before the results above, we ran 510 headless agent sessions across three separate benchmark designs, and almost none of it survived scrutiny. The oracles we chose (gopls, tsc, go build) could only grade statically resolvable questions inside a single repository, which is the same class of question a text search answers well. Requiring a mechanical oracle had silently selected the questions our baseline was already good at. Two of six answer keys turned out to be wrong, and in both contested cases the agents had been more correct than the key grading them. The one arm that genuinely exercised the tool was an arm we invented, which stripped the agent of grep and shell, and which describes a configuration nobody runs.

What survived was a defect detector: the exercise found a real bug in Enola's own config parser that had gone undetected across a 72-repository corpus, because it was invisible on any repository with a single package.

The write-up names all six ways the harness lied, including a control group that silently inherited a running instance of the tool under test.

Read: three MCP server benchmarks and why each one failed →

The full results, and how to reproduce them.

Read docs/BENCHMARKS.md

How these benchmarks are built, and what each one caught →

Compare Enola with other code-graph tools →