How we test Enola against 91 repositories.

Enola extracts a repository into a graph and grades what a change did to it. Extractor changes often move that graph in places the author did not expect: a Ruby fix lands, the unit tests pass, and a route count drops by seven in a repository nobody was looking at.

For that reason, unit tests are only one part of the release check. The full corpus covers 91 public open-source repositories: 448,086 source files, 8,194,786 facts, and 26 language tags. A sweep indexes every repository once cold and twice warm, for 273 runs. The results are published on the benchmarks page.

Why fixtures are not enough

Unit tests check the cases we anticipated. They are less useful for interactions we did not know to model, especially those involving repository layout, configuration, caches, and multiple languages.

An extractor can pass every fixture and still read a Rails scope constraints: { format: :html } block as a /html path prefix, reporting two hundred routes one segment deeper than they are served. It can also resolve a markdown link by consulting files left by a previous run, so the same commit produces one graph cold and another warm. Both defects depend on context outside the narrow case covered by a fixture.

The corpus tests that wider context by indexing real repositories repeatedly and comparing the results.

The eight questions

Together, the scripts cover reproducibility, regression precision, resource use, coverage, architecture labels, command contracts, agent behaviour, and changes between sweeps.

1. Reproducibility across cold and warm runs

A diff between two snapshots is useful only if the snapshots are reproducible. Every repository is therefore indexed three times, once cold and twice warm. The script compares the receipt's snapshot_id and the SHA-256 of facts.jsonl across all three runs. Comparing cold with warm tests that cached and from-scratch runs agree.

Current result: 91 of 91 repositories, 273 runs, zero drift. Reaching that result required fixing two bugs. Markdown links were resolved by stat-ing the filesystem, so this repository's own docs linked to nothing on a cold run and to something on the next. The output directory was also ignored only at the repository root. A cluster snapshot left an .enola directory in each subdirectory, and Enola indexed its own generated context file as source. Ten repositories in the corpus carried those extra facts, including the Linux kernel.

The first version of the script also created its own failure. It used a custom output directory, .enola-bench, while the default ignore globs hard-code .enola/**. On the next Discourse run, the benchmark indexed its own output as source. The defect was real, but the harness—not normal usage—had triggered it.

Two rules came out of that, and both are written at the top of the scripts because each one silently invalidated a full sweep before it was understood:

• Run a binary with no mcp-arch.yaml beside it. Enola resolves its default configuration relative to the executable's directory, and a config's extractor list replaces the built-in default rather than merging with it. A binary built inside the Enola repository picks up that repository's checked-in config for every codebase it analyses. The scripts copy the binary into a neutral bin/ for that reason.
• Use the default output directory. See above. Run what a user runs.

2. Reporting only new regressions

Enola's CI output should describe what a change introduced without repeating every existing finding in the repository. Otherwise the result is too noisy to use as a gate. bench-ratchet tests that behaviour directly.

The ratchet pins a baseline and puts twenty repositories through four states each. Files are added, never edited, so every revert is a delete:

• No change. Expect PASS and exactly zero added facts. This is the noise floor, and without it a PASS on a real change means nothing.
• A benign addition. A new leaf module. Expect PASS, with the delta naming exactly the two or three facts added.
• An injected cycle. Two new modules importing each other. Expect FAIL, exactly one regression, and none of the repository's pre-existing findings repeated.
• Reverted. Expect PASS again. The verdict is a function of the tree, not of history.

All four states behave as expected on all twenty repositories, across ten languages. Together, those repositories hold 1,620 pre-existing findings, up to 235 in one repository. The injected cycle produces exactly one regression and repeats none of them.

The runs use --fail-on=cycles. Without an explicit policy, Enola fails nothing by design and every state exits 0. The script states the flag in its header so the FAIL result is not mistaken for default behaviour.

Swift is absent from the table because a Swift module is a declared SPM target. Two added directories that import each other form no edge and no cycle. Injecting a cycle would require editing Package.swift, which would make the reverted state a file restore rather than the same delete used for the other languages.

3. Calibrating the memory ratchet

bench-mem-ratchet compares a sweep with pinned ceilings for peak heap and allocations per fact.

It started out grading run 1 only, for a reason that looked solid: the cold run parses every file instead of reading cached facts, and allocates roughly twice as much, so cold must be the expensive path. The first measurement falsified it. The warm runs peaked higher, on both ceilinged repositories: the Linux kernel at 6,537 MiB cold against 6,594 and 6,731 warm.

Peak depends on how much memory is live at once, not on total allocations. A warm run allocates about half as much, but it holds the decoded extractor cache alongside the facts and may collect less often. Two later sweeps showed that neither pass consistently peaks higher: one sweep peaked cold on both repositories; the next peaked warm on one and cold on the other.

The ratchet therefore grades two metrics with different tolerances:

At v264, the sweep crossed both ceilings: peak heap reached 8,504 MiB against an 8,000 MiB ceiling, and allocation churn reached 505.9 against 480. We reran the old and new binaries cold and back to back to reduce the effect of machine state. Peak heap did not reproduce, but churn on the kernel rose from 459.8 to 500.2—about ten times the metric's observed spread.

The increase appeared across multiple extractors and was not proportional to output. Ruby and TypeScript repositories with no C showed it, while C#, Java, Rust and F# stayed flat. gmsh was the clearest row: +0.5% facts, +27.5% allocations. Diffed heap profiles pointed to one mechanism. Node.Child() in the tree-sitter binding allocates a Go object on every call, and 96% of the added allocations arrived through it. Three passes added over three extractor versions each walked the syntax tree again in Go to find a handful of nodes, allocating for every node visited along the way.

Every visited node has to cross into Go, so changing the traversal idiom does not remove the cost. Matching inside tree-sitter does: the scan runs in C and only matches cross the boundary. On one 92 KB file, that reduced allocations from 32,150 to 845 and ran 3.2× faster. The shipped version beat the version where the regression was found on both memory metrics while preserving the new facts. All 90 comparable repositories produced byte-identical fact streams before and after.

An earlier set of ceilings exposed a second problem. We measured them with a two-repository sweep, then applied them to a full 81-repository run. By the time the full run reached those repositories, the machine had been under load for forty minutes. The mismatched conditions failed a binary that used less peak memory than its predecessor in a back-to-back comparison. We now use restricted runs to compare changes, but set ceilings only from the same kind of run they will grade.

4. The inverted Android taxonomy

The layers explainer labels a repository's architecture—for example, android-clean or dotnet-clean—and reports violations against that taxonomy's layer order. For a long time, the corpus recorded the output without checking whether the label and order were correct.

The Android taxonomy shipped with its layer order inverted. It reported three violations against Google's reference application, the project from which the taxonomy had been derived. The sweep, ratchet, and coverage checks all stayed green because they checked extraction, not the architectural conclusion built on top of it.

bench-arch now grades thirty repositories against an expectations file that records what the project itself documents: nowinandroid's own architecture guide, eShop's reference Clean Architecture layout, Discourse as a Rails application that also ships an Ember front end. Twenty-four of thirty pass at the current extractor version, and no recorded gap has moved.

Two details in the expectations file are worth calling out.

null is an explicit expectation. Where a project claims no architecture, a null expected pattern means Enola must report nothing. A false positive is still a mismatch.

Ten rows record missing ecosystem support. No taxonomy exists for those ecosystems, so reporting nothing is the known gap rather than a pass. The first version of the script printed ten green lines for those ten unsupported repositories. Those rows close when Enola starts naming an architecture, the reverse of an ordinary null expectation.

5. Known gaps and cross-repository coverage

Six of the thirty architecture rows currently do not pass. Each carries a known_gap naming the planned work, which changes the result from FAIL to KNOWN without hiding it. If a known gap begins to pass, the script highlights it so the expectation can be promoted to a graded row.

Keeping these rows in the file makes missing coverage visible alongside the taxonomies that already work.

The cross-repository coverage report likewise includes resolved and unresolved edges. Without the unresolved count, the number of resolved edges has no useful denominator. For the Nextcloud server and two first-party apps indexed into one graph, the result is:

  service      classification  detected  resolved  unresolved
  collectives  coverage_gap          2         0           2
  contacts     isolated              0         0           0
  server       coverage_gap         13         0          13

None of the thirteen server edges resolves. The report classifies server as coverage_gap rather than isolated, because "depends on nothing" and "Enola could not tell" look identical from the graph alone. Nextcloud apps reach the server through in-process PHP APIs rather than cross-repository HTTP, so the linker has little to match. The cluster remains in the corpus as a record of that limitation.

6. Commands the corpus does not exercise

Sweep, ratchet and coverage all measure extraction and the delta over it. A whole command surface (constraints, plan, endpoint, architecture history, the provider seam) produces no facts of its own, so a green sweep says nothing about any of it.

A regression found while validating 0.4.0 lived in that surface. A plain file named enola at a repository root made the constraints loader return ENOTDIR, aborting the snapshot. A unit test caught it, but nothing in the benchmark harness would have.

bench-newsurface checks command contracts rather than extracted facts, so it produces no results file. It runs against throwaway clones under a temporary root because plan --patch and the history commands write to the repository. Running them against the corpus would alter the input to the next sweep.

7. Does the graph help an agent?

bench-variance asks six structural questions of a coding agent, five trials each, across six arms that differ only in how the tool is offered: no tool at all, MCP attached, the rules file that enola install really writes, the session hooks, the instruction placed in the user's own message, and one arm with text search removed so the agent can only discover paths through the graph. Every answer key was built before any arm ran. The most recent run is 180 sessions over two hours.

The graph did not improve the measured answers.

On these six questions, an agent restricted to the graph answered as well as an agent with grep and cost about 1.5× as much. The run produced two other findings.

Agents rarely reached for the structural tool on their own: 3 of 120 sessions across four voluntary channels. In this run, the installer ran for real, a canary verified that the rules reached the model, and the hook fired in all thirty of its sessions. Tool use became consistent only when text search was removed. The result still applies to this harness and its system prompt; we have not run the cross-harness control.

Agents are stably wrong, not randomly wrong. The first run assumed that asking an agent the same structural question five times would produce varying answers. Instead, all thirty question-by-arm cells returned five byte-identical answer sets from independent sessions spread over three hours. Every arm answered two of the six questions identically wrong in every session. Repeating the question did not expose the incomplete answer.

The script has nevertheless found product defects. One run exposed a TypeScript extraction bug that broke impact analysis and traversal for monorepos using the common @scope/pkg convention. The bug did not appear on a single-package repository and had survived a 72-repository corpus sweep. After the fix, F1 on the affected question rose from 0.60 to 1.00, self-agreement from 0.721 to 1.000, and median cost fell from $4.07 to $0.50. With an empty graph, the agent had spent more than twenty turns reading candidate files and reconstructing the missing structure by hand.

A second gap remains open. The graph under-surfaces implicit relationships such as Go interface satisfaction by embedding, especially in the presence of cross-repository distractors. On one question, the graph arm surfaced the contested file in 1 of 5 trials; the text-search arms found it in all 5.

F1 also missed one difference between the arms. On a contested question, the text-search arms never found a particular file. The graph session found it, considered it, and excluded it. Both received the same score because the metric compares answer sets, not the evidence considered along the way.

8. Comparing fact kinds between sweeps

bench-compare diffs two sweeps by repository, language, and fact kind. Aggregate totals can hide a regression in one category behind gains in another.

Fact counts are expected to move after an extractor change. The comparison therefore flags a decrease in any fact kind or language even when the total rises. Release 0.4.0 added roughly 6,000 facts to GitLab while dropping seven routes. Those seven were false positives caused by Apollo directives being read as GraphQL fields.

More facts are not necessarily better. A Rails route reader that invents six verbs per declared resource raises the count while reducing accuracy; correcting it took the corpus from 10,776 Rails routes to 7,963. Missing facts can also create false findings elsewhere. Before Enola could read Rust utoipa attribute routes, the domain explainer reported nineteen components on crates.io as calling outbound third-party endpoints. Those endpoints were actually served by the Rust half of the same repository. Once the routes were visible, all nineteen findings disappeared and call-site resolution rose from 2 of 91 to 90 of 91.

Where this sits in the loop

Not every script runs after every change. The sweep accepts an ENOLA_REPOS list, so we usually test an extractor change against the two or three repositories it could affect plus the two used for memory ceilings. That takes minutes. Before a release, we run the full corpus and two sweeps forty minutes apart. The separated sweeps can expose drift hidden when three runs share a process lifetime and warm page cache.

The variance run is the exception: it costs two hours and real money, so it runs when something about the agent-facing surface changes, not on a schedule.

What we apply elsewhere

1. Pin a baseline. Questions about a change need a recorded before value, not whatever state happens to be on disk.
2. Measure the noise floor. The no-change case must produce exactly zero added facts before a changed case is meaningful.
3. Check conclusions as well as counts. Reproducible extraction did not catch the inverted Android taxonomy.
4. Match calibration and grading conditions. A ceiling taken from a short run should not grade a full sweep.
5. Record known gaps. Keep unsupported repositories and unresolved edges visible so improvements and regressions both change the report.

All eight scripts and the corpus definitions are part of Enola's development workflow. The benchmarks page shows the latest sweep, so its numbers move as the extractors change.

Related reading

• What a code graph actually costs in memory. The numbers the memory ratchet protects, from a small service to the Linux kernel.
• Why static analysis is hard: an extractor cache changelog. The sibling discipline, one version number at a time.
• Benchmarks. The current published results, per repository and per language.

curl -fsSL https://raw.githubusercontent.com/enola-labs/enola/main/install.sh | sh