Corpus

corpus

Corpus extractor -- public symbols + their embeddable text.

Ported from the POC docstring_corpus.py, but rebranched onto axm-ast (tree-sitter) instead of the stdlib ast module: symbols, signatures, and docstrings now come from axm_ast.core.parser and the first docstring line from axm_ast.docstring_parser.parse_docstring.

The unified embedding text (AC3) is, per symbol::

Text Only

embed_text = docstring if present else normalized code/signature

so a public documented symbol contributes its intent while an undocumented one still contributes its code shape -- one engine, one vector space, the registry picked per symbol.

Symbol dataclass

A single public symbol projected into the corpus.

Source code in packages/axm-echo/src/axm_echo/corpus.py

@dataclass(frozen=True)
class Symbol:
    """A single public symbol projected into the corpus."""

    qualname: str
    name: str
    package: str
    workspace: str
    kind: str  # "function" | "class"
    signature: str
    doc_first_line: str
    doc_full: str
    body_norm: str
    path: str
    line: int

    @property
    def has_doc(self) -> bool:
        """Whether the symbol carries a non-empty docstring."""
        return bool(self.doc_full.strip())

    @property
    def embed_text(self) -> str:
        """Unified embed text: docstring if present, else code/signature."""
        if self.doc_full.strip():
            return f"{self.signature}\n{self.doc_full}".strip()
        return f"{self.signature}\n{self.body_norm}".strip()

    def as_dict(self) -> SymbolDict:
        """Flat dict view (qualname, package, signature, embed_text, ...)."""
        return {
            "qualname": self.qualname,
            "name": self.name,
            "package": self.package,
            "workspace": self.workspace,
            "kind": self.kind,
            "signature": self.signature,
            "doc_first_line": self.doc_first_line,
            "doc_full": self.doc_full,
            "body_norm": self.body_norm,
            "embed_text": self.embed_text,
            "has_doc": self.has_doc,
            "path": self.path,
            "line": self.line,
        }

embed_text property

Unified embed text: docstring if present, else code/signature.

has_doc property

Whether the symbol carries a non-empty docstring.

as_dict()

Flat dict view (qualname, package, signature, embed_text, ...).

Source code in packages/axm-echo/src/axm_echo/corpus.py

def as_dict(self) -> SymbolDict:
    """Flat dict view (qualname, package, signature, embed_text, ...)."""
    return {
        "qualname": self.qualname,
        "name": self.name,
        "package": self.package,
        "workspace": self.workspace,
        "kind": self.kind,
        "signature": self.signature,
        "doc_first_line": self.doc_first_line,
        "doc_full": self.doc_full,
        "body_norm": self.body_norm,
        "embed_text": self.embed_text,
        "has_doc": self.has_doc,
        "path": self.path,
        "line": self.line,
    }

discover_package_roots()

Discover every package directory across the configured scope.

Walks each workspace root from :func:axm_echo.scope.load_scope, covering both the <ws>/packages/<pkg> convention and the flat other/<pkg> layout. The set of roots is data-driven (no frozen package list), so newly added packages are picked up automatically (AC4).

Returns:

Type	Description
list[Path]	Sorted, de-duplicated package directories.

Source code in packages/axm-echo/src/axm_echo/corpus.py

def discover_package_roots() -> list[Path]:
    """Discover every package directory across the configured scope.

    Walks each workspace root from :func:`axm_echo.scope.load_scope`,
    covering both the ``<ws>/packages/<pkg>`` convention and the flat
    ``other/<pkg>`` layout. The set of roots is data-driven (no frozen
    package list), so newly added packages are picked up automatically
    (AC4).

    Returns:
        Sorted, de-duplicated package directories.
    """
    seen: set[Path] = set()
    roots: list[Path] = []
    for workspace_root in load_scope():
        for pkg in _packages_in_workspace(workspace_root):
            if pkg not in seen:
                seen.add(pkg)
                roots.append(pkg)
    return sorted(roots)

extract_monorepo()

Extract public symbols across every discovered package (corpus).

Returns:

Type	Description
list[SymbolDict]	The concatenated symbol corpus for the configured scope.

Source code in packages/axm-echo/src/axm_echo/corpus.py

def extract_monorepo() -> list[SymbolDict]:
    """Extract public symbols across every discovered package (corpus).

    Returns:
        The concatenated symbol corpus for the configured scope.
    """
    out: list[SymbolDict] = []
    for pkg in discover_package_roots():
        out.extend(extract_package(pkg))
    return out

extract_package(pkg_root)

Extract every public function/class of a package via axm-ast.

"Public" follows the axm-ast convention: a symbol exported in the module's __all__ when present, else any module-level name without a leading underscore. Test files and empty __init__ files are skipped; unparseable files are silently ignored.

Parameters:

Name	Type	Description	Default
pkg_root	Path	Package directory (containing src/<pkg>/ or a flat source tree).	required

Returns:

Type	Description
list[SymbolDict]	One Symbol per public function/class, each carrying
list[SymbolDict]	qualname, signature, doc_first_line, doc_full,
list[SymbolDict]	body_norm and the derived embed_text.

Source code in packages/axm-echo/src/axm_echo/corpus.py

def extract_package(pkg_root: Path) -> list[SymbolDict]:
    """Extract every public function/class of a package via axm-ast.

    "Public" follows the axm-ast convention: a symbol exported in the
    module's ``__all__`` when present, else any module-level name without
    a leading underscore. Test files and empty ``__init__`` files are
    skipped; unparseable files are silently ignored.

    Args:
        pkg_root: Package directory (containing ``src/<pkg>/`` or a flat
            source tree).

    Returns:
        One ``Symbol`` per public function/class, each carrying
        ``qualname``, ``signature``, ``doc_first_line``, ``doc_full``,
        ``body_norm`` and the derived ``embed_text``.
    """
    package, workspace = _package_meta(pkg_root)
    symbols: list[Symbol] = []
    for py in _iter_source_files(pkg_root):
        try:
            mod = extract_module_info(py)
        except (OSError, ValueError):
            continue
        mod_qual = _module_qualname(py, pkg_root)
        symbols.extend(_module_symbols(mod, mod_qual, py, package, workspace))
    return [s.as_dict() for s in symbols]