Skip to content

Counter

counter

Token counting with tiktoken.

count(text, model='o200k_base')

Return the token count for text.

Uses tiktoken with model encoding. Falls back to len(text) // 4 when tiktoken is unavailable.

Source code in packages/axm-smelt/src/axm_smelt/core/counter.py
Python
 8
 9
10
11
12
13
14
15
16
17
18
19
20
def count(text: str, model: str = "o200k_base") -> int:
    """Return the token count for *text*.

    Uses tiktoken with *model* encoding.  Falls back to ``len(text) // 4``
    when tiktoken is unavailable.
    """
    try:
        import tiktoken

        enc = tiktoken.get_encoding(model)
        return len(enc.encode(text))
    except Exception:  # noqa: BLE001
        return len(text) // 4