Cluster

cluster

Cross-package echo clustering + anti-signals (v7).

Ported from the dedup-detection POC poc_c3_docstring_pairs.py: instead of comparing function bodies (code <-> code) it compares their promises -- the public docstrings -- across the whole monorepo, surfaces cross-package pairs that are semantically close, and groups them into clusters.

The raw cross-package pairs are then split by the v7 anti-signals so the duplicate clusters carry signal, not noise:

• trivial accessors -- a getter/setter promise (Return the name.) is boilerplate intent shared by every model; filtered up front (the POC's -96% noise cut).
• parallel API (AS2a/AS2b by name) -- a pair whose names start with their own package token (excel_export / word_export) is parallel-by-design, not copy-paste; demoted to the parallel_api bucket.
• boilerplate frequency -- a docstring first-line that recurs across many symbols (CLI entry point.) matches all its twins at ~1.0 without meaning duplication; demoted to the boilerplate bucket.

CALIBRATION (E3bis / AXM-2171, refine-loop, 2026-06): the two boilerplate seuils were tuned against the real monorepo corpus (3.6k documented symbols). min_repeat=4 catches the genuine mass-recurring promises (CLI entry point. 9x, main entry point. 5x, render the failure-path ... 10x, the office table ops 4x) while leaving the 550+ unique terse promises and the ground-truth duplicates (RateLimitError, request_with_retry) alone; the length floor stays at MIN_DOC_CHARS=15 so a legitimate terse promise (APIUnavailable, 38 chars) is never re-dropped. Two consecutive passes are stable. The defaults below are the calibrated values, not raw POC ports.

cluster_pairs(pairs, *, max_cluster_size=MAX_CLUSTER_SIZE)

Group pairs into connected components (clusters of echoing symbols).

Each duplicate pair is an edge; the connected components of the resulting graph are the echo clusters. Returns one sorted index list per cluster, largest first, so a triple of mutually-duplicate symbols lands in a single cluster rather than three disjoint pairs.

A component with more than max_cluster_size members is dropped: such a mega-cluster is a union-find over-merge (a structural-conformity signal), not a genuine duplicate echo. The bound is paramétrable; the default is :data:MAX_CLUSTER_SIZE.

Parameters:

Name	Type	Description	Default
pairs	list[Pair]	Duplicate (i, j, score) edges.	required
max_cluster_size	int	Reject any component strictly larger than this.	MAX_CLUSTER_SIZE

Returns:

Type	Description
list[list[int]]	One sorted index list per surviving cluster, largest first.

Source code in packages/axm-echo/src/axm_echo/cluster.py

def cluster_pairs(
    pairs: list[Pair], *, max_cluster_size: int = MAX_CLUSTER_SIZE
) -> list[list[int]]:
    """Group pairs into connected components (clusters of echoing symbols).

    Each duplicate pair is an edge; the connected components of the resulting
    graph are the echo clusters. Returns one sorted index list per cluster,
    largest first, so a triple of mutually-duplicate symbols lands in a single
    cluster rather than three disjoint pairs.

    A component with more than ``max_cluster_size`` members is **dropped**: such
    a mega-cluster is a union-find over-merge (a structural-conformity signal),
    not a genuine duplicate echo. The bound is paramétrable; the default is
    :data:`MAX_CLUSTER_SIZE`.

    Args:
        pairs: Duplicate ``(i, j, score)`` edges.
        max_cluster_size: Reject any component strictly larger than this.

    Returns:
        One sorted index list per surviving cluster, largest first.
    """
    parent: dict[int, int] = {}

    def find(x: int) -> int:
        parent.setdefault(x, x)
        root = x
        while parent[root] != root:
            root = parent[root]
        while parent[x] != root:
            parent[x], x = root, parent[x]
        return root

    def union(x: int, y: int) -> None:
        parent[find(x)] = find(y)

    for gi, gj, _score in pairs:
        union(gi, gj)

    groups: dict[int, list[int]] = defaultdict(list)
    for node in parent:
        groups[find(node)].append(node)
    clusters = [
        sorted(members)
        for members in groups.values()
        if len(members) <= max_cluster_size
    ]
    clusters.sort(key=len, reverse=True)
    return clusters

cross_pairs(matrix, packages, *, threshold)

Cross-package candidate pairs above threshold (brute-force matmul).

Compares every embedded row against every other in batched dot products, keeping only the upper triangle and only pairs whose two symbols live in different packages (same-package echoes are out of scope).

Parameters:

Name	Type	Description	Default
matrix	NDArray[float64]	(n, dim) row-normalized embedding matrix.	required
packages	list[str]	Per-row package name, parallel to matrix rows.	required
threshold	float	Minimum cosine to keep a pair.	required

Returns:

Type	Description
list[Pair]	(i, j, cosine) triples with i < j and distinct packages.

Source code in packages/axm-echo/src/axm_echo/cluster.py

def cross_pairs(
    matrix: NDArray[np.float64], packages: list[str], *, threshold: float
) -> list[Pair]:
    """Cross-package candidate pairs above ``threshold`` (brute-force matmul).

    Compares every embedded row against every other in batched dot products,
    keeping only the upper triangle and only pairs whose two symbols live in
    *different* packages (same-package echoes are out of scope).

    Args:
        matrix: ``(n, dim)`` row-normalized embedding matrix.
        packages: Per-row package name, parallel to ``matrix`` rows.
        threshold: Minimum cosine to keep a pair.

    Returns:
        ``(i, j, cosine)`` triples with ``i < j`` and distinct packages.
    """
    pairs: list[Pair] = []
    n = matrix.shape[0]
    batch = 256
    for start in range(0, n, batch):
        end = min(n, start + batch)
        block = matrix[start:end] @ matrix.T
        for li in range(block.shape[0]):
            gi = start + li
            row = block[li].copy()
            row[gi] = 0.0
            for raw in np.where(row >= threshold)[0]:
                gj = int(raw)
                if gj <= gi or packages[gi] == packages[gj]:
                    continue
                pairs.append((gi, gj, float(row[gj])))
    return pairs

generic_docs(symbols, *, min_repeat=4)

First-lines recurring across >= min_repeat symbols are boilerplate.

LESSON (POC, 2026-06): a brute length filter wrongly dropped a legitimate terse promise (APIUnavailable, 38 chars). Corpus FREQUENCY is the right signal -- a first-line shared by many symbols is boilerplate, a unique terse line is not. Length stays only as a floor (MIN_DOC_CHARS).

Source code in packages/axm-echo/src/axm_echo/cluster.py

def generic_docs(symbols: list[SymbolDict], *, min_repeat: int = 4) -> set[str]:
    """First-lines recurring across ``>= min_repeat`` symbols are boilerplate.

    LESSON (POC, 2026-06): a brute length filter wrongly dropped a legitimate
    terse promise (``APIUnavailable``, 38 chars). Corpus FREQUENCY is the right
    signal -- a first-line shared by many symbols is boilerplate, a unique terse
    line is not. Length stays only as a floor (``MIN_DOC_CHARS``).
    """
    counts: dict[str, int] = defaultdict(int)
    for sym in symbols:
        key = _doc_first_line(sym).lower()
        if key:
            counts[key] += 1
    return {k for k, c in counts.items() if c >= min_repeat}

is_parallel_api(a, b)

Whether the pair is a parallel API surface, not a duplicate (AS2a/AS2b).

A symbol whose name starts with its own package token (excel_export in axm-excel, word_export in axm-word) is parallel-by-design. If either side of the pair matches its package token, demote the pair.

Source code in packages/axm-echo/src/axm_echo/cluster.py

def is_parallel_api(a: SymbolDict, b: SymbolDict) -> bool:
    """Whether the pair is a parallel API surface, not a duplicate (AS2a/AS2b).

    A symbol whose name starts with its own package token (``excel_export``
    in ``axm-excel``, ``word_export`` in ``axm-word``) is parallel-by-design.
    If either side of the pair matches its package token, demote the pair.
    """
    for sym in (a, b):
        short = str(sym.get("package", "")).removeprefix("axm-")
        toks = str(sym.get("name", "")).lstrip("_").split("_")
        if short in _WS_TOKENS and toks and toks[0] == short:
            return True
    return False

is_trivial_accessor(sym)

Whether sym is a trivial getter/setter accessor (filtered, AC3).

Trivial accessors are the dominant noise source in cross-package docstring matching: every model declares Return the name. and they all match each other at ~1.0 without being duplicate behaviour. We detect them on the available projection -- an accessor-shaped docstring first line, or an accessor symbol name paired with a terse promise.

Source code in packages/axm-echo/src/axm_echo/cluster.py

def is_trivial_accessor(sym: SymbolDict) -> bool:
    """Whether *sym* is a trivial getter/setter accessor (filtered, AC3).

    Trivial accessors are the dominant noise source in cross-package
    docstring matching: every model declares ``Return the name.`` and they
    all match each other at ~1.0 without being duplicate *behaviour*. We
    detect them on the available projection -- an accessor-shaped docstring
    first line, or an accessor symbol name paired with a terse promise.
    """
    first = _doc_first_line(sym)
    if _ACCESSOR_DOC_RE.match(first):
        return True
    name = str(sym.get("name", ""))
    return name in _ACCESSOR_NAMES and len(first) < MIN_DOC_CHARS * 2

split_pairs(pairs, symbols, generic)

Split raw cross-package pairs into (dupes, parallel, boilerplate).

Applies the v7 anti-signals: a pair whose both sides are boilerplate (generic first-line or trivially short doc) is boilerplate; a name-parallel pair is parallel API; everything else is a genuine duplicate candidate.

Source code in packages/axm-echo/src/axm_echo/cluster.py

def split_pairs(
    pairs: list[Pair], symbols: list[SymbolDict], generic: set[str]
) -> SplitPairs:
    """Split raw cross-package pairs into (dupes, parallel, boilerplate).

    Applies the v7 anti-signals: a pair whose *both* sides are boilerplate
    (generic first-line or trivially short doc) is boilerplate; a name-parallel
    pair is parallel API; everything else is a genuine duplicate candidate.
    """
    dupes: list[Pair] = []
    parallel: list[Pair] = []
    boilerplate: list[Pair] = []
    for gi, gj, score in pairs:
        a, b = symbols[gi], symbols[gj]
        if _both_boilerplate(a, b, generic):
            boilerplate.append((gi, gj, score))
        elif is_parallel_api(a, b):
            parallel.append((gi, gj, score))
        else:
            dupes.append((gi, gj, score))
    return dupes, parallel, boilerplate