Text length versus tokens: choose the tokenizer explicitly

Characters are not tokens

A tokenizer can split a word into pieces or combine common byte sequences. Character count and word count are different measurements.

There is no fixed characters-per-token ratio that is exact across languages, code and punctuation. Measure the actual text.

Text: hello world
cl100k_base: 2 tokens
JavaScript UTF-16 length: 11

Select the encoding

Nav Tools exposes o200k_base, cl100k_base and p50k_base. The same text can have different counts under different vocabularies.

Confirm the target system's tokenizer rather than selecting the smallest count. A model name alone is not a substitute for its documented encoding.

Keep protocol overhead separate

The tool counts the text you supply. Chat role wrappers, tool definitions, images and provider-specific framing are not included automatically.

Use the actual service's usage fields for billing and compare like-for-like payloads when investigating a discrepancy.

Inspect count changes after editing

Whitespace, line endings and punctuation can alter token boundaries. A shorter character string does not always guarantee fewer tokens.

Use Prompt Diff to inspect the edit and Token Counter for both versions with the same encoding. The displayed token ID list is limited to the first 1,000 entries.