enola/ Blog/ Apache Airflow Architecture Audit

Blog

We asked enola to audit Apache Airflow's architecture. Here's what a staff engineer would notice.

Coupling hotspots, cyclic dependencies, god classes, complexity outliers — and a 56,308-node blast radius hiding inside one module.

Before a staff engineer signs off on a refactor, they do an in-depth analysis every time: where's the coupling concentrated, are there cycles or layering problems, which functions are too complex to touch safely, and, critically, does the risk live at the module level or hide inside one or two files. We pointed enola at Apache Airflow's monorepo and walked through that exact review.

One generate_snapshot call, 9.5 seconds, 112,103 facts across 11,869 files in Python, TypeScript, Java, and OpenAPI specs. Everything below is what came out of querying that graph.

The snapshot

shell
enola generate_snapshot /path/to/airflow
MetricValue
Generation time9.5s
Files parsed11,869
Facts extracted112,103
Symbols63,688 (40,801 methods, 12,181 functions, 8,554 classes, 1,383 types)
Routes6,342
Dependencies39,517 (17,775 internal, 12,374 stdlib, 9,368 external)
Storage facts64 (DB tables/models)
Modules2,492
Cyclic dependency chains26
Languages detectedPython, TypeScript, Java, OpenAPI

This is one generate_snapshot call. Everything below comes from querying the resulting graph.

First pass: where does everything funnel through?

The first thing a staff engineer checks isn't bugs, it's concentration. Which modules does the rest of the codebase route through, regardless of what feature you're touching?

ModuleFan-inFan-outBlast radius
airflow-core/src/airflow/models1,38531256,308
airflow-core/src/airflow/utils1,0236962,421
airflow-core/src/airflow (root)8673966,730
providers/common/compat1,209150,692
devel-common/test_utils1,4478535,564

models is the data layer — DAG, TaskInstance, DagRun, the ORM definitions. utils is the shared utility belt: logging, config, timezone handling. Both are exactly what you'd expect to top this list in a mature platform, and that's the point of checking it first: high fan-in here isn't a surprise finding, it's confirmation the codebase shape matches its job.

The number worth sitting with is providers/common/compat, a backward-compatibility shim layer with 1 outgoing dependency and 1,209 incoming ones. That's a module whose entire job is being depended on. Healthy, as long as nobody starts adding logic to it.

Second pass: known anti-patterns

With enola you get the signal of codebase health. The count alone doesn't tell you what to do, though. Here's each signal with the actual next step.

SignalFoundWhat it means
Cyclic dependencies26 chainsModules with circular import relationships
Layer violationsincluded in cycle/layer scanCode reaching across layers it shouldn't
God classes286Symbols with disproportionate dependent counts
Deepest dependency chain57 levelsFound in test modules, e.g. tests/unit/api_fastapi/core_api/routes
Complexity outliers15 flagged, top score 69SchedulerJobRunner (69), WatchedSubprocess (61)

Cyclic dependencies: triage by boundary, not by count

26 isn't a number to clear in one PR, it's a number to sort. Pull the actual cycle list (query_facts with relation: imports, or explore on each flagged module) and split it two ways: cycles contained entirely inside one package versus cycles that cross a package boundary that's supposed to be one-directional — e.g. a providers/* package importing back into airflow-core internals it shouldn't depend on. The second kind is the real risk; the first kind is usually two tightly-coupled internal files that happen to import each other and rarely causes a real bug.

For each boundary-crossing cycle: find the specific back-edge import creating the loop, then check whether it resolves with dependency inversion. Pull the shared type or interface both sides need into a third module neither side needs to import the other for. Add a CI import-linter rule pinned to the current count as a baseline so it can't grow silently, and ratchet it down over time instead of trying to zero it out at once.

Layer violations: fix the ones that'll get copied

Enola's layer detection flags imports that don't match the architectural pattern it found (for api_fastapi: routes/services/datamodels/). For each flagged edge, there are only two outcomes: either the layering rule is right and the import needs to go through the proper layer, or the "rule" is too strict for a legitimate exception and it's not actually a violation. Sort by where it sits: a violation in an actively-developed area like core_api is worth fixing now, because the next few PRs will copy whatever pattern they see there, violation or not. A violation in a stable, rarely-touched corner can wait.

God classes: rank by dependents × churn, not by dependents alone

286 is too many to fix one at a time, and most of them don't need fixing — a stable symbol with high fan-in that nobody's actively touching is low risk by definition. Rank instead by dependent count multiplied by how often the file actually changes. The ones that are both heavily depended on and frequently modified are where the actual bugs happen. For the top 5–10 by that combined score, the standard move is extracting a narrower interface — most callers don't need the whole symbol's surface, just a slice of it. Shrinking what callers depend on shrinks the practical blast radius even before the implementation changes. Track the rest; don't try to clear the list.

Deep dependency chains: a build-speed problem here, not a risk problem

The 57-level chain is confirmed test-only. That reframes it: this isn't "something might break," it's "incremental test runs are slower than they need to be, and there's more flakiness surface than there should be." Deep test chains usually come from test files importing other test files importing other test files, rather than a deliberate layering decision. The fix is flattening: pull shared setup into one fixtures module 1–2 levels deep that everything imports directly, instead of importing it transitively through other tests. Lower priority than the first three.

Complexity outliers: prioritize by graph position, not by score alone

These are the most actionable items on the list because the fix is well-understood: extract named, independently-testable sub-steps until each piece is back under a sane per-function complexity threshold. But don't just work down the score list. SchedulerJobRunner at 69 is the top priority not because it's the highest score, it's because it sits inside jobs/, the highest-blast-radius runtime path in the repo. WatchedSubprocess at 61 is second: also runtime-critical, in the task SDK's lifecycle management, slightly more contained. A complexity-69 function in a leaf utility module would rank lower than either of these despite the higher score, because almost nothing depends on what it touches.

Before decomposing either: run impact_analysis scoped to the function itself, not the module, to know exactly what depends on its current behavior. Then write characterization tests for that behavior before splitting it up. This is precisely the kind of function where "I refactored this and broke something three layers away" happens, because high complexity and high blast radius tend to show up together.

Third pass: check beyond the module-level number

This is the step a quick scan skips and a real review doesn't. models/ shows 1,385 direct dependents and a 56,308-node blast radius as a module. But "the models module" is over 30 files, and the risk is not evenly spread:

FileImported byRisk
taskinstance.py63 filescritical
dagrun.py60 filescritical
dag.py29 fileshigh
serialized_dag.py~15 filesmedium
xcom.py~12 filesmedium
log.py~5 fileslow
task_store.py~3 fileslow

A 12x gap between the riskiest and safest file in the same directory is the entire reason module-level fan-in is a starting point, not a verdict. taskinstance.py reaches into the scheduler, three separate api_fastapi route and service files, the entire serialization layer, settings.py, policies.py, the task SDK's type definitions, and 50+ provider test files. log.py touches five files, full stop. Treating both as "part of the high-risk models module" would be the wrong call in both directions — overcautious about log.py, underprepared for taskinstance.py.

Of the module's 614 symbols, 183 are internal and unexported. Those carry zero blast radius by definition and can be refactored freely. The other 431 are where impact_analysis earns its keep — run per-symbol, not per-module.

What a staff engineer would actually recommend

  1. Stop reasoning about models/ as one unit. Run impact_analysis on the specific class or function you're touching. DagRun and Log are both "in models" and have nothing else in common, risk-wise.
  2. Watch the serialization boundary. The 187-symbol serialization module mirrors model shapes directly. A field rename or type change there isn't just a code change, it needs a coordinated update plus a database migration, since 64 of the storage facts are tables this module touches.
  3. Test coverage already points at the right place. tests/unit/models (955 symbols) is the largest test module in the repo and the primary validation layer for this exact risk area; tests/unit/jobs (411 symbols) and the api_fastapi test suites catch the integration fallout one layer out.
  4. Prefer additive changes over breaking ones in models/ and utils/ — like new fields, new methods, not renames or signature changes — given how far a breaking change here actually reaches.

Verdict

airflow-core/src/airflow/models is the highest-risk module to refactor in the codebase: 1,385 direct dependents, a transitive blast radius covering over half the 112,103-fact graph, and the kind of fan-in where a single renamed field can surface as a failure three layers and several packages away. That's not a flaw in Airflow's design — a platform this size needs a small number of modules everything else can rely on. It's a fact to plan around: anything touching models/ or utils/ needs impact_analysis run first, scoped to the exact symbol, not the directory.

Why an agent can't reconstruct this picture

Agents and enola are architecturally different. impact_analysis is a graph traversal: a deterministic walk over parsed facts, executed by an algorithm, returning the same 56,308 nodes every time it's run against the same codebase. An agent searching and reading files isn't running that traversal. It's generating a plausible-sounding answer from whatever subset of the codebase fit in its context window, with the gaps filled in by pattern-matching.

That gap shows up two ways. Completeness: a 56,308-node blast radius means dependency chains traced many hops deep, across package boundaries, including callers that only reach a type through one of its methods rather than naming it directly. Agents do not hold that much code in context at once, so they cannot have actually traced all of it. And if they report anything, it's not deterministic. Hallucination: when a piece of the codebase isn't known to an agent, it doesn't say "unknown." It generates the most plausible-sounding dependency based on naming patterns and prior training. It may seem correct, given it phrases it confidently, specifically, with no way to tell that output apart from a verified fact by reading the response alone.

Enola reports 56,308 dependents on models/ because it walked 56,308 real edges in a graph built from real parsers. There's no equivalent claim available to an agent working from search and reads. Only an estimate.

Try it on your own repo

shell
curl -fsSL https://raw.githubusercontent.com/enola-labs/enola/main/install.sh | sh
shell
"Generate an architectural snapshot of /path/to/repo."
"Where is coupling concentrated, and are there any cyclic dependencies or layer violations?"
"What would break if I changed [your highest-risk module]? Show me the impact analysis, broken down by file."

enola on GitHub — Apache 2.0, free, local. Full tool reference in ARCHITECTURE.md.

FAQ

What is the riskiest module to refactor in Apache Airflow?

Based on an enola architecture audit, airflow-core/src/airflow/models carries the highest refactor risk in the codebase: 1,385 direct dependents and a transitive blast radius of 56,308 nodes, more than half the repository's full dependency graph. Within that module, risk is concentrated in a few files — taskinstance.py (63 dependents) and dagrun.py (60 dependents) — while others in the same directory, like log.py (about 5 dependents), are comparatively safe to change.

Does Apache Airflow have circular dependencies or other technical debt signals?

An enola scan found 26 cyclic dependency chains, 286 god classes (symbols with a disproportionate number of dependents), and dependency chains up to 57 levels deep in some test modules. The most complex function in the codebase is SchedulerJobRunner, in the core scheduling module, with a cyclomatic complexity score of 69.

How do you do a pre-refactor architecture audit?

A pre-refactor audit typically checks where dependency coupling is concentrated (fan-in/fan-out per module), scans for cyclic dependencies and layer violations, flags unusually complex functions, and then drills from the module level down to the file or symbol level, since module-level risk numbers can hide large gaps between individual files. Tools that build a deterministic dependency graph, such as enola, can compute this from a single snapshot rather than requiring it to be re-derived by hand for each question.

Can you run this kind of audit with an AI agent instead of a dependency graph?

Not reliably. AI agents are prone to hallucinations. They make assumptions and even when checking the repository multiple times, they summarise the information in a non-deterministic way. A pre-built dependency graph answers these questions with the same answer each time.

enola is open source under Apache 2.0 — free, local, no data leaves your machine.

shell
curl -fsSL https://raw.githubusercontent.com/enola-labs/enola/main/install.sh | sh