Words do not always become one model token. The way scripts, mixed languages and specialized text are segmented can change costs, truncation behavior and reliability.
Characters, words and tokens are not synonyms
Many language systems process text after it has been split into token units, which may represent whole words, fragments, spaces or punctuation. The exact tokenizer depends on the system and version. Two strings with the same character count can use different token counts, and short text in one language can require substantially different segmentation from similarly sized text in another. Avoid promising a universal ratio between words and tokens.
Why this matters outside billing
Segmentation can affect how a system handles spelling, names, inflected forms, code-switching and long documents. An instruction requiring an exact character count may be violated when the model predicts tokens instead of counting characters. Digit sequences, unusual legal abbreviations and non-Latin scripts deserve verification rather than assumptions of perfect copy accuracy. These are potential failure modes, not proof that every multilingual model fails the same way.
The truncation trap
A support team may upload multilingual terms and conditions to a context-limited service. If text is chunked or truncated without preserving headings, a clause can lose its exception. A translated answer may then confidently omit a condition. Before production, record the actual input sizes in the supported tokenizer, confirm whether the complete source reached the model, and check that retrieved chunks retain cross-references and dates.
Test scripts and registers separately
Build evaluation cases for the languages that users actually employ, including regional spelling, right-to-left scripts where relevant, numbers, dates, local names and mixed-language messages. Have qualified native-language reviewers assess meaning preservation, not simply fluency. Test both directions of translation. A grammatically polished translation can still reverse negation or change the legal force of ‘may’ and ‘must’.
A usable quality gate
For multilingual customer answers, set independent pass criteria for names and numbers, citation alignment, intent recognition and appropriate refusal. Include examples with informal speech and typos. Track performance by language without interpreting small sample differences as population-level ranking. Escalate high-stakes material to an expert when the evidence or translation remains ambiguous.
Commercial and regional caution
Token rates, limits, supported languages and region restrictions depend on a particular provider, model and tier at a particular date. None are asserted here. Obtain dated first-party documentation from the relevant product team before estimating a bill or declaring a language supported. An evaluation of one version does not validate a successor.
What to retain
Record the exact tokenizer or service version, sample inputs, observed token counts, truncation settings and human-review findings. Re-run the suite on model and prompt changes. These artifacts allow teams to distinguish text representation problems from retrieval failures and ordinary factual hallucination.
Sources & Further Reading
- NIST AI 600-1 Generative AI Profile — NIST.
- Holistic Evaluation of Language Models — Stanford CRFM.
- AILuminate benchmarks and methodology — MLCommons.
Primary Scope and applicability vary by task and jurisdiction. This article is educational guidance, not legal, medical or financial advice.