Estimates the number of tokens in a text string using a byte-based heuristic. Uses UTF-8 byte length divided by 4, which accounts for multi-byte characters (emojis, CJK, etc.) that typically consume more tokens than ASCII text.
The text to estimate token count for
Estimated token count
Estimates the number of tokens in a text string using a byte-based heuristic. Uses UTF-8 byte length divided by 4, which accounts for multi-byte characters (emojis, CJK, etc.) that typically consume more tokens than ASCII text.