Skip to content

Counter

counter

Token counting with tiktoken.

CounterBackend

Bases: StrEnum

Backend used to produce a token count.

Currently only :attr:TIKTOKEN exists; the enum is retained as the seam for a future HuggingFace/SentencePiece backend (Llama/Mistral/Gemma).

Source code in packages/axm-smelt/src/axm_smelt/core/counter.py
Python
26
27
28
29
30
31
32
33
class CounterBackend(StrEnum):
    """Backend used to produce a token count.

    Currently only :attr:`TIKTOKEN` exists; the enum is retained as the seam
    for a future HuggingFace/SentencePiece backend (Llama/Mistral/Gemma).
    """

    TIKTOKEN = _TIKTOKEN_VALUE

count(text, model='o200k_base')

Return the token count for text.

Uses tiktoken with model encoding; a claude* or unknown model is routed to the o200k_base proxy.

Source code in packages/axm-smelt/src/axm_smelt/core/counter.py
Python
73
74
75
76
77
78
79
80
def count(text: str, model: str = "o200k_base") -> int:
    """Return the token count for *text*.

    Uses tiktoken with *model* encoding; a ``claude*`` or unknown model is
    routed to the ``o200k_base`` proxy.
    """
    n, _ = count_with_backend(text, model)
    return n

count_with_backend(text, model='o200k_base')

Return (token_count, backend) for text.

Always counts with tiktoken (backend :attr:CounterBackend.TIKTOKEN): a claude* or otherwise unknown model is routed to the o200k_base proxy encoding rather than a len // 4 approximation.

Source code in packages/axm-smelt/src/axm_smelt/core/counter.py
Python
57
58
59
60
61
62
63
64
65
66
67
68
69
70
def count_with_backend(
    text: str, model: str = "o200k_base"
) -> tuple[int, CounterBackend]:
    """Return ``(token_count, backend)`` for *text*.

    Always counts with tiktoken (backend :attr:`CounterBackend.TIKTOKEN`):
    a ``claude*`` or otherwise unknown model is routed to the ``o200k_base``
    proxy encoding rather than a ``len // 4`` approximation.
    """
    enc = _ENC.get(model)
    if enc is None:
        enc = _resolve_encoding(model)
        _ENC[model] = enc
    return len(enc.encode(text)), CounterBackend.TIKTOKEN