Skip to content

Fix pipeline — deterministic test-suite reorganiser

The axm audit_fix command and the audit_fix MCP tool (AuditFixTool in axm_audit.tools.audit_fix) drive a deterministic pipeline that batch-relocates, splits, merges and renames test files to satisfy the AXM 3-level pyramid + canonical naming conventions. It consumes findings from three rules:

  • TEST_QUALITY_PYRAMID_LEVEL (unit vs integration vs e2e)
  • TEST_QUALITY_FILE_NAMING (test_{symA}__{symB}.py for integration/e2e)
  • TEST_QUALITY_NO_PACKAGE_SYMBOL (out-of-pipeline; surfaced for manual review)

The pipeline runs in dry-run by default and mutates only on --apply. It targets Python test trees. It does not reorganise JS/TS tests.

Use safely

Bash
axm audit_fix . --json-output
axm audit_fix . --rules '["TEST_QUALITY_FILE_NAMING"]' --json-output

The rules list selects naming/pyramid stages; unknown IDs are not rejected. In apply mode helper extraction and Ruff polish still run independently of that stage filter. An empty list is therefore not a general no-op guarantee.

A dry-run plans one pass and skips the apply-only non-canonical-tier and layout-flattening stages. It is not a complete simulation of all apply effects. Inspect warnings as well as operations. Run tests before and after applying, and compare collection, outcomes and coverage yourself.

Pipeline architecture

Text Only
0.5 NON-CANONICAL-RELOCATE  tests/functional/*  → tests/integration/
0.  FLATTEN                 heterogeneous Test* classes → top-level funcs
1.  RELOCATE                PYRAMID_LEVEL mismatch → git mv across tiers
1.5 FLATTEN_LAYOUT          tests/<tier>/<subdir>/ → flat layout
2.  SPLIT                   FILE_NAMING verdict=SPLIT     → anvil moves units
3.  COLLIDE / MERGE         FILE_NAMING verdict=COLLIDE   → anvil moves units
4.  RENAME                  FILE_NAMING verdict=NAME_MISMATCH → git mv

The whole pipeline runs inside a fixed-point loop (MAX_ITERATIONS=6), since each mutation can expose new findings the audit could not see on the previous iteration. Iteration stops early when a pass emits zero ops. In dry-run mode the loop runs exactly once (no mutation).

NO_PACKAGE_SYMBOL findings are out-of-pipeline — the verdict is context-dependent (legitimate formal check vs. candidate for deletion) and surfaced in a separate report section pointing the user to /scenario-rename or manual inspection.

Stage 0.5 RELOCATE moves files under any non-canonical tier (tests/functional/, tests/hooks/, ...) into tests/integration/ so that subsequent stages only ever see canonical paths; CANONICAL_TIERS in models.py is the allow-list (unit, integration, e2e). tests/fixtures/ is excluded (the _NON_TEST_DIR_NAMES set in layout_and_move.py) — by AXM convention it holds static test data (corpora, snapshots, baselines), not test files.

Module layout (hexagonal split)

The applicator lives at src/axm_audit/core/fix/, modules organised by hexagonal layer:

Text Only
core/fix/
├── __init__.py             — public API: run, format_report, PipelineReport, FileOp, OpKind
├── models.py               — FileOp, OpKind, PipelineReport + constants
│                             (NON_DETERMINISTIC_RULES, CANONICAL_TIERS, MAX_ITERATIONS, TOP_K)
├── io_primitives.py        — cst_load/save/top_level/unwrap + _git_mv
├── paths.py                — tier_for_path, retier, safe_filename,
│                             module_path_for_test_file, file_depth_from_project
├── tests_ast.py            — read-only AST: tests, classes (pathological detection),
│                             helpers, markers (usefixtures), imports analysis
├── cst_rewrite.py          — write CST: flatten class, rename, delete, reorder,
│                             depth patch (__file__), imports (insert/dedupe/
│                             backfill) + project import index cache
├── findings.py             — audit ingestion, canonical filename, collect_unfixable
├── layout_and_move.py      — relocate_non_canonical_tiers (Stage 0.5),
│                             flatten_tier_layout (Stage 1.5),
│                             _rewrite_cross_test_imports,
│                             _safe_move_units (wraps anvil),
│                             _resolve_helper_conflicts / _resolve_conftest_shadowing
├── stages_plan.py          — plan_flatten / plan_relocate / plan_naming (pure)
├── stages_execute.py       — _execute_flatten/_relocate/_rename/_split/_merge
│                             + execute() dispatcher
├── extract_helpers.py      — post-pipeline helper extraction to
│                             tests/<tier>/_helpers.py or conftest.py
├── pipeline.py             — run() + fixed-point loop + _ruff_format_tests
└── report.py               — format_report CLI output

Dependency layers

Text Only
report          → models
pipeline        → models, stages_plan, stages_execute, extract_helpers,
                  layout_and_move, cst_rewrite, findings
extract_helpers → cst_rewrite, io_primitives, paths, tests_ast
stages_execute  → cst_rewrite, findings, io_primitives, layout_and_move,
                  models, paths, tests_ast
stages_plan     → findings, models, paths, tests_ast
layout_and_move → cst_rewrite, io_primitives, models, paths, tests_ast
findings        → models, paths, tests_ast, (lazy: stages_plan)
cst_rewrite     → io_primitives, paths, tests_ast
tests_ast       → (stdlib only)
paths           → (stdlib only)
io_primitives   → libcst
models          → (stdlib only)

One lazy cycle: findings.collect_unfixable → stages_plan.plan_flatten (needed to surface pathological FILE_NAMING cases the pipeline can't auto-fix).

Behavioral notes

  1. Anvil's rename= param doesn't help cross-file collisions — it validates target absence under the original name before applying the rename. The pipeline works around it by renaming the symbol in source first (via rename_top_level_in_source in cst_rewrite.py), then handing anvil a clean conflict-free move.

  2. Caches and file moves — cache invalidation matters after in-flight mutations. Residual findings are collected separately from the mutation stages; a warning or missing residual result is not proof of parity.

  3. if TYPE_CHECKING: imports are invisible to anvil — MockerFixture imported only inside that block is treated as not-imported when the symbol using it (only via type annotation) is moved. _backfill_missing_imports() in cst_rewrite.py walks if TYPE_CHECKING: blocks and reproduces the wrapper at the target.

Edge cases

  • _make_pkg signature drift across files — _resolve_helper_conflicts() (layout_and_move.py) renames the source-side helper to H__from_<stem> when source and target define helpers with the same name but different bodies. shared_helpers="duplicate" then copies cleanly.
  • @pytest.mark.usefixtures("X") invisible to anvil — anvil walks AST refs on moving symbols, but marker arguments are string literals, so fixtures injected via marker used to stay in source and disappear when source was stripped. _collect_marker_fixtures_to_move() (tests_ast.py) now scans moving units for usefixtures markers and adds the referenced fixtures to the move list when they're source-defined and not target-visible.
  • Decorator-referenced module-level constants dropped on SPLIT — a test decorated with @_alias where _alias = pytest.mark.skipif(...) is a top-level Assign used to produce un-collectable target files: anvil moved the test but left _alias (and any constants it referenced like CASES) behind, then _finalize_split_anchor git-mv-renamed source to the anchor target with the moved constants already stripped from source — so even the anchor lost them. _collect_module_level_deps_to_copy() (tests_ast.py) computes the transitive closure of free-name references inside decorators of the moving units, restricted to top-level Assign / AnnAssign defs in source. _copy_module_level_deps_to_target() (layout_and_move.py) splices those statements into the target as text after anvil runs, without touching source — so the anchor's source-rename keeps the constants too. Closure follows chains (B = A + 1 → also carry A = 1), preserves source order, and is surgical: unrelated top-level names are not carried.
  • SPLIT / MERGE module-level load-time order — when a moving unit references names from its decorator chain (e.g. def helper(): ... + CONST = helper() + _alias = pytest.mark.skipif(not CONST, ...) + @_alias @pytest.mark.parametrize("c", CONST)), the combination of anvil's shared_helpers="duplicate" mode and _copy_module_level_deps_to_target can leave the target with top-level statements in an order that breaks module load (_alias = ... not CONST ... evaluating before CONST = helper(), or CONST = helper() evaluating before def helper). _topological_reorder_decorator_deps() (layout_and_move.py) runs as the last step of _safe_move_units and is target-centric: it builds a top-level dependency graph from the target tree alone, spanning Assign / AnnAssign / FunctionDef / AsyncFunctionDef / ClassDef. Free Name(Load) references are walked in Assign.value / AnnAssign.value+annotation and FunctionDef.decorator_list / ClassDef.decorator_list+bases+keywords — bodies of defs are NOT load-evaluated and are intentionally skipped. The hoist set is the transitive closure of names referenced by any module-level decorator; hoisted statements are topologically sorted (deps before dependents) and reinserted before the first remaining statement that load-time-references any hoisted name. The pre-anvil source tree is used only as a stable tiebreaker between valid linearisations — not as the source of truth for ordering, which matters when a target receives content from multiple FileOps (split-with-multiple-targets, merge, or both): no single source tree describes all contributors. Statements outside the hoist set keep their existing relative position — the pass is surgical and idempotent (running it twice on a correctly-ordered target is a no-op).
  • Path(__file__).parents[N] drift after relocate — a file moved from tests/unit/core/test_X.py (4-deep) to tests/integration/test_X.py (3-deep) keeps its FIXTURES = Path(__file__).parents[2] / "fixtures" constant, which now points to project root instead of tests/. _patch_file_dunder_depth() (cst_rewrite.py) detects the depth delta from file_depth_from_project() and rewrites both surface forms (parents[N] subscript and .parent.parent... chains) via libcst. The id-collection pre-pass MUST run after _DunderPatcher since libcst rebuilds nodes during any visit (even no-op transforms), so original module ids are stale by the time _PatchChainOnce runs.
  • Conftest fixture shadowing on MERGE — when source has no local fixture X (relies on conftest's body) but target has a local X with a different body, the moved tests bind to target's local at runtime and fail with Symbol not found. _resolve_helper_conflicts also renames the source-side helper when target lacks it BUT a conftest on target's ancestor chain provides a fixture of the same name. _collect_conftest_fixtures() (tests_ast.py) walks the chain.
  • Helper-rename misses transitive references — _resolve_helper_conflicts used to check helper collisions only against names directly referenced by the moving units. A test consuming a fixture which itself calls a helper would miss the helper at rename time and bind to target's same-named but body-different version. Fix: extend referenced with a fixed-point closure that walks names → source helpers → names-they-reference until no new helpers appear.
  • Promoted helper's dependencies left behind — when extract_shared_helpers_in_tier() (extract_helpers.py) promotes a fixture to tests/conftest.py, its helper dependencies (already living in tests/<tier>/_helpers.py) are not imported by the destination conftest. _synth_import_from_helpers() in cst_rewrite.py is a last-resort backfill in _backfill_missing_imports() that scans every tests/<tier>/_helpers.py for a top-level def of the missing name and synthesises a from tests.<tier>._helpers import <name> statement.
  • _git_mv silent overwrite — the shutil.move fallback used when git mv refused (target exists) used to destroy pre-existing files. _git_mv() in io_primitives.py now raises FileExistsError on pre-existing targets; _execute_relocate / _execute_rename (stages_execute.py) re-route through _safe_move_units (via the _reroute_through_safe_move helper). The pre-call existence check is exists() (not is_file()) so a pre-existing directory at the target doesn't slip through.
  • _git_mv race when two ops target the same destination in one iteration — shutil.Error("Destination path already exists") is caught and translated to FileExistsError, then re-routed through _safe_move_units (same path as the silent-overwrite case above).
  • _retier ate the .py extension for tests at tests/ root — for tests/test_X.py (no tier subdir yet) the function used to do parts[1] = target_lvl and return a directory path, then _safe_move_units crashed with IsADirectoryError on ast.parse(target.read_text()). paths.py:retier now branches on len(parts) == 2: inject the tier between tests and the file instead of substituting at index 1.
  • Overlong renamed identifiers (E501) — _bounded_rename in _safe_move_units (layout_and_move.py) falls back to a 6-char sha1 digest of the stem (__from_<digest>) when the verbose form would push the def line past 88 chars. Stem hashing keeps cross-source-file uniqueness; the verbose form is preferred when it fits.
  • Split docstring placement (E402) — SPLIT seeds a new file with a module docstring before anvil prepends imports. _reorder_module_statements (cst_rewrite.py) used to treat any leading ast.Expr/Constant/str as part of the import head, leaving the docstring after the imports (E402 trigger on every subsequent import). It now detects the docstring as a separate statement (PEP 257: must be the very first body element to count as one) and unconditionally promotes it to position 0 in the rewritten body.

Convergence and rollback limits

Apply makes at most six iterations and may stop earlier when a pass emits zero operations. Reaching the cap is not proof that every finding was cleared. A follow-up dry-run and real test run are caller-owned checks. Mixed-tier files that cannot be relocated unanimously may remain for review.

The implementation snapshots the literal tests/ directory to a temporary directory. Exceptions during the guarded mutation/polish block or a failed syntax gate trigger restoration of that directory.

This is a scoped rollback, not a repository transaction:

  • The snapshot and gate do not cover tests_axm_*/, the Git index or files outside tests/. Moves can affect Git staging.
  • If no tests/ existed, there is no snapshot to restore.
  • The gate compiles tests/**/test_*.py; it does not import modules, resolve fixtures, run pytest collection or execute tests.
  • Warnings are not necessarily exceptions and do not universally trigger rollback.
  • Residual/unfixable collection occurs after the guarded apply block.

A concrete collection failure can survive the syntax gate: renaming a file to tests/integration/test_add.py when tests/unit/test_add.py already exists can cause pytest's default import mode to report an import-file mismatch. Use the project's intended pytest import policy (AXM uses --import-mode=importlib) and verify actual collection/execution after apply.

Do not use the tool's success or the applied flag as evidence of parity, idempotence or guaranteed convergence. Review the actual diff and rerun tests.

Out of pipeline (agent-driven follow-ups)

The slash names below refer to separately installed operator skills, not commands or entry points distributed by axm-audit. Manual review against the reported scenario is always possible.

  • TEST_QUALITY_DUPLICATE_TESTS → /dedup-tests
  • TEST_QUALITY_PRIVATE_IMPORTS → /private-imports-clear
  • TEST_QUALITY_PYRAMID_LEVEL (residual mixed-tier files) → /pyramid-relocate
  • TEST_QUALITY_TAUTOLOGY → /tautology-clear
  • PRACTICE_TEST_MIRROR → /mirror-fix
  • TEST_QUALITY_NO_PACKAGE_SYMBOL → /scenario-rename or manual deletion review.