Tokenization Explained: How Large Language Models Break Text Into Tokens

2026-07-09 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Novice, short

Summary

Tokenization is the fundamental process by which Large Language Models (LLMs) break down raw text into smaller, meaningful units called tokens for processing. While humans effortlessly parse words, computers require explicit tokenization to determine unit boundaries. The article explains that tokens can represent complete words, parts of words, single characters, punctuation, numbers, or special symbols. It details the limitations of simple space-based splitting, character-level tokenization (leading to long sequences and slow training), and word-level tokenization (resulting in massive vocabularies). Modern LLMs predominantly employ subword tokenization, which splits uncommon words into reusable pieces like "un" + "believe" + "able," significantly reducing vocabulary size while maintaining flexibility. The choice of tokenizer critically impacts input sequence length, vocabulary size, memory usage, training/inference speed, and overall model performance.

Key takeaway

For Machine Learning Engineers designing or fine-tuning Large Language Models, understanding tokenization is critical. Your choice of tokenizer directly influences vocabulary size, sequence length, and ultimately, model training and inference efficiency. Opting for subword tokenization strategies, like those used in modern LLMs, can significantly optimize resource usage and performance. Be aware that different tokenizers will process the same text into varying token counts, impacting your model's input and output consistency.

Key insights

Subword tokenization is crucial for LLMs, balancing vocabulary size and text representation efficiency.

Principles

• Tokenization is a core LLM design choice.
• Subword tokenization balances vocabulary and flexibility.
• Different tokenizers yield different token counts.

Method

Modern LLMs use subword tokenization, splitting uncommon words into reusable pieces to reduce vocabulary size while representing diverse text.

In practice

• Punctuation marks are often treated as separate tokens.
• Uncommon words like "unbelievable" split into subwords.

Topics

• Tokenization
• Large Language Models
• Subword Tokenization
• Vocabulary Size
• Natural Language Processing
• Model Performance

Best for: AI Student, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Your LLM Has Never Read a Single Word — How Tokenization Grinds Text Into Numbers — LLMs process numeric tokens, not words or letters, leading to inherent limitations like character blindness and variable processing costs across languages.
• What is Tokenization? All you need to know — Tokenization is a critical NLP decision impacting model efficiency, robustness, and language representation, not just a preprocessing step.
• Word predictability estimates from language models are not robust to tokenizer vocabulary — Language model predictability estimates and reading time predictions are significantly impacted by tokenizer vocabulary size, particularly for words split into subword units.
• In-Place Tokenizer Expansion for Pre-trained LLMs — In-place tokenizer expansion efficiently upgrades LLMs for new languages, reducing token fragmentation and improving decode speed without quality loss.
• Inside the Mind of a Large Language Model — LLMs convert human language into numerical representations through preprocessing, tokenization, and vocabulary mapping before learning.
• AI 101: What Is a Token (and why it runs AI)? — Tokens are the fundamental units AI models process, impacting context, performance, and cost.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.