Persian tokenizer efficiency
How many tokens each model spends on the same meaning in Persian versus English. Measured, not estimated.
The gap between the best and worst tokenizer on this board
A Persian speaker pays between 1.14× and 4.41× more per idea than an English speaker, depending only on which model they use.
What this measures
One narrow, deterministic thing: the number of tokens each model's tokenizer spends encoding the same meaning in Persian and in English. That is a cost fact and a context-window fact. Persian text is billed by the token, so a tokenizer that spends four times as many tokens on Persian makes the same conversation four times as expensive and fills the context window four times as fast.
What this does not measure
It is not a measure of model quality, Persian fluency, reasoning, or usefulness. A model can tokenize Persian efficiently and still write it badly. This board is named for the one axis it measures, because a ranking that implies more than it tested is worse than no ranking.
Measured 2026-08-19. Re-run when a significant model ships.
what English costswhat Persian costs on top
| Rank | Model | Vendor | Persian cost | Persian tokens | English tokens | Vocabulary | Tokenizer file |
|---|---|---|---|---|---|---|---|
| 1 | PersianMindPersian-specific model | University of Tehran | 839 | 737 | 41510 | universitytehran/PersianMind-v1.0 | |
| 2 tied | Gemma | 908 | 692 | 256000 | Xenova/gemma-tokenizer✻ | ||
| 2 tied | Gemma 2 | 908 | 692 | 256000 | Xenova/gemma2-tokenizer✻ | ||
| 4 | Mistral Small 3 | Mistral AI | 955 | 698 | 131072 | mistralai/Mistral-Small-24B-Instruct-2501 | |
| 5 tied | Llama 3 | Meta | 959 | 689 | 128256 | Xenova/llama-3-tokenizer✻ | |
| 5 tied | Llama 3.1 | Meta | 959 | 689 | 128256 | Xenova/Meta-Llama-3.1-Tokenizer✻ | |
| 7 tied | GPT-4o | OpenAI | 969 | 689 | 200000 | Xenova/gpt-4o✻ | |
| 7 tied | gpt-oss 20B | OpenAI | 969 | 689 | 200019 | openai/gpt-oss-20b | |
| 9 | GLM-4.5 | Z.ai | 1191 | 689 | 151365 | zai-org/GLM-4.5 | |
| 10 | DeepSeek-V3 | DeepSeek | 1203 | 693 | 128815 | deepseek-ai/DeepSeek-V3 | |
| 11 tied | Qwen2.5 | Alibaba | 1777 | 689 | 151665 | Qwen/Qwen2.5-7B-Instruct | |
| 11 tied | Qwen3 | Alibaba | 1777 | 689 | 151669 | Qwen/Qwen3-8B | |
| 13 tied | GPT-4 / GPT-3.5 Turbo | OpenAI | 2216 | 689 | 100263 | Xenova/gpt-4✻ | |
| 13 tied | Phi-4 | Microsoft | 2216 | 689 | 100352 | microsoft/phi-4 | |
| 13 tied | OLMo 2 | Allen Institute for AI | 2216 | 689 | 100278 | allenai/OLMo-2-1124-7B | |
| 16 tied | Mistral v3 | Mistral AI | 3018 | 723 | 32768 | Xenova/mistral-tokenizer-v3✻ | |
| 16 tied | Maral 7BPersian-specific model | MaralGPT | 3018 | 723 | 32000 | MaralGPT/Maral-7B-alpha-1 | |
| 18 | Claude | Anthropic | 3053 | 698 | 65000 | Xenova/claude-tokenizer✻ | |
| 19 | Grok-1 | xAI | 3054 | 692 | 131072 | Xenova/grok-1-tokenizer✻ |
Models sharing one tokenizer
These ship byte-identical merge tables, so they are one measurement listed under each name rather than separate results.
- Llama 3.1 — Llama 3
- Qwen3 — Qwen2.5
Why this is worth measuring
Persian is under-represented in almost every tokenizer vocabulary, and the cost of that lands entirely on Persian speakers: higher bills, shorter usable context, and slower responses for the same thought. Almost nobody publishes the number. Making it public is the point of this page.
Why you can trust the numbers
- Round-trip. A model whose token sequence does not decode back to the exact input is dropped from the board, not reported with a caveat. A segmentation that cannot be reversed is not a segmentation of that text.
Method
Every passage in the paired corpus is encoded with each model's own tokenizer file, read from HuggingFace at the exact commit recorded in the table. Counts are exact. Nothing here calls a model, so the whole board costs nothing to reproduce and cannot drift with a provider's mood.
The corpus
28 passages written for this purpose, each in Persian with a faithful English translation, across registers from news and technical writing to legal, conversational and literary prose. Both sides carry the same information, because a word-for-word gloss would inflate one language for grammatical reasons and corrupt the ratio. The passages are original compositions, so no model can have memorised them from a public benchmark.
Reproduce it
The harness, the corpus and the results are all in the repository. This fetches each tokenizer file, re-measures every passage and rewrites the table:
pnpm run bench:tokenizerOpen data
Not on the board
These tokenizers need an authenticated request. They are named here rather than quietly omitted, because a board that hides what it could not reach is not showing its own weakness.
- google/gemma-3-4b-it
- CohereLabs/aya-expanse-8b
- PartAI/Dorna-Llama3-8B-Instruct
- utter-project/EuroLLM-9B