Regression isolation · agent outcome · reproducibility
We tested whether Enola isolates a newly introduced cycle, whether an agent corrects it when the result returns automatically, and whether the underlying graph is reproducible. The gate runs enforced --fail-on=cycles — Enola fails nothing unless a policy names it, so that flag is what the FAIL column measures.
Measured August 23, 2026 on 91 public repositories and controlled fixtures. Read the methodology and reproduce the results.
The August 23 benchmark corpus contains 8,014,394 facts across 91 repositories and 26 measured language tags, with zero parse errors across 444,986 files parsed.
| Repository | Language | Files | Facts | Cold | Warm |
|---|---|---|---|---|---|
| linux | C | 55,883 | 1,908,734 | 155.2s | 40.1s |
| dart-sdk | Dart | 16,823 | 453,013 | 97.2s | 14.1s |
| gitlab | Ruby | 55,936 | 544,431 | 38.0s | 22.6s |
| runtime | C# | 23,547 | 529,820 | 96.3s | 62.2s |
| rust | Rust | 37,565 | 404,094 | 35.2s | 14.0s |
| spark | Scala | 5,745 | 219,732 | 58.3s | 40.0s |
| grafana | Go | 11,343 | 186,365 | 14.4s | 8.7s |
| supabase | TypeScript | 7,336 | 74,645 | 29.1s | 3.5s |
| airflow | Python | 4,363 | 73,820 | 7.4s | 4.6s |
9 of 91 shown. Full corpus on GitHub →
Same commit, three runs, identical snapshot ID and fact hash — on all 91 repositories.
One sweep indexed all 91 repositories once cold and twice warm; every one reproduced, covering 8,014,394 facts with zero drift. The sweep was then run a second time, forty minutes later, and all 81 fact streams are byte-identical between the two runs as well — a stronger result than the in-sweep check, because two sweeps share neither a process lifetime nor a warm page cache. A diff between snapshots is meaningful only when the snapshots themselves are reproducible.
Pin a baseline on a real repository, move it through four controlled states, and check whether Enola isolates the one new regression without repeating existing findings.
1. No change — run the check again with no code changes. The expected result is exactly zero new facts, edges, or findings.
2. Benign addition — drop in a new file that doesn't depend on or get depended on by anything. The result names exactly the 2–3 facts added; it isn't reported as a regression.
3. Injected cycle — add two files that import each other, a genuine architectural mistake. This is the one thing that should fail — even in a repo already carrying 100+ older issues.
4. Reverted — delete those two files. The verdict returns to passing, showing that it is derived from the current tree rather than the previous result.
| Repository | Pre-existing findings | No change | Injected cycle | Reverted |
|---|---|---|---|---|
| gitea | 171 | PASS | FAIL · 1 | PASS |
| enola | 145 | PASS | FAIL · 1 | PASS |
| superset | 133 | PASS | FAIL · 1 | PASS |
| cognee | 121 | PASS | FAIL · 1 | PASS |
| jellyfin | 103 | PASS | FAIL · 1 | PASS |
| gitbucket | 50 | PASS | FAIL · 1 | PASS |
All four states came back correctly on 20 repositories across ten languages. Those repositories contained 1,839 pre-existing findings, up to 239 in one repository; none were repeated as new. Of 9,131 findings across the full corpus, 1,298 (14.2%) reach confidence 1.0 — the floor a named policy gates at — so even with every explainer named, that is the share eligible to fail a build. The rest are estimates and stay advisory whatever you configure.
Five times the findings underneath, the same verdict on top, and 22 of 22 newly introduced entries correctly classified.
An August 2026 experiment added three deeper analyzers to the same ratchet. The risk was that the extra advisory volume would leak into the verdict and make the gate harder to trust. The experiment ran the same four repository states with and without those analyzers.
| State | New analyzer advisories | Filed as incidental | What they were |
|---|---|---|---|
| 1. No change | 0 | 0 | The analyzers add nothing to a no-op |
| 2. Benign addition | 9 | 7 | All nine name the injected function itself |
| 3. Injected cycle | 4 | 2 | All four are the injected cycle, as recursion or dead interfaces |
| 4. Reverted | 0 | 0 | Back to the floor |
The classification rule. An entry counts as correct only if it is attributable to the injected code, or correctly filed as incidental — a pre-existing symbol that moved because a capped list re-ranked when the new symbol entered it, not because the change touched it. All 22 met one of the two, and the 4,836 pre-existing findings stayed out of the verdict entirely.
Two analyzers, one defect. The injected cycle is two modules calling each other, so it is also mutual recursion: the cycles explainer and the performance analyzer reach it independently. Corroboration, not duplication.
Where the precision comes from. Only the high-confidence orphan tier is written onto facts — 1,602 of 12,678 computed candidates. Medium and low stay as leads, because type usage and dynamic dispatch are not tracked as edges, so a symbol can look unreferenced while being used as a field type. Putting those into a diff would be a false positive dressed as a finding.
What it costs you. A helper added in one commit and called in the next carries a dead-code advisory on that first commit. It is a true positive — the function really is unreferenced — but it is one you may not want. It stays advisory, and the exit code does not move.
Measured August 2, 2026 on 12 of the ratchet repositories, in a separate run from the corpus sweep above — which is why the comparison count here is 984 pre-existing findings rather than the 1,839 the current ratchet reports. That run has not been repeated since; the ratchet has grown to 20 repositories and the explainers have changed under it, so read these figures as a dated analyzer experiment rather than a current product comparison. n = 22 new entries: a small sample, and stated as one.
Across eleven committed cross-language fixtures, does Enola resolve the connection—and report what remains unresolved?
The current benchmark also includes real Nextcloud, Bitwarden, and Dart/Flutter clusters. It reports both resolved and unresolved call sites, distinguishing a genuinely isolated service from one with a coverage gap. Full cluster results →
Carried forward from the earlier benchmark run; sections above were re-run for the August 8 corpus.
Same coding task, given to Claude Code headless three ways. The obvious implementation closes a dependency cycle — does the agent catch it before calling the job done?
Bare — no MCP server, no instruction, no hooks: just the agent and the task.
Instruction only — Enola’s MCP server is available and the agent is instructed to pin a baseline and check its own change. The agent must initiate the check.
Enola loop — the same setup, plus a Stop hook that runs the comparison automatically when the agent tries to finish.
"The regression wassrc/domain ↔ src/store: my first attempt made the domainimport { listOrders }from the store, which closed a loop with the store's pre-existingimport type { Order }from the domain. I removed the offending edge."— unprompted, from a loop-arm transcript
9 trials total using Claude Code 2.1.205 in headless mode. Small sample — read it as proof the mechanism works end to end, not as a measured effect size. No arm called the comparison tool directly; the loop arm succeeded because the Stop hook ran the check automatically, not because the agent remembered to ask.
Every page like this one shows you the measurements that survived. Here are the ones that did not, and why.
Before the results above, we ran 510 headless agent sessions across three separate benchmark designs, and almost none of it survived scrutiny. The oracles we chose — gopls, tsc, go build — could only grade statically resolvable questions inside a single repository, which is the same class of question a text search answers well. Requiring a mechanical oracle had silently selected the questions our baseline was already good at. Two of six answer keys turned out to be wrong, and in both contested cases the agents had been more correct than the key grading them. The one arm that genuinely exercised the tool was an arm we invented, which stripped the agent of grep and shell, and which describes a configuration nobody runs.
What survived was a defect detector: the exercise found a real bug in Enola's own config parser that had gone undetected across a 72-repository corpus, because it was invisible on any repository with a single package.
The write-up names all six ways the harness lied, including a control group that silently inherited a running instance of the tool under test.