What is an LLM? Tokens, Embeddings, and the Big Picture

2026-06-14 · Source: MLWhiz: Recs|ML|GenAI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

This analysis details the fundamental architecture of Large Language Models (LLMs), defining them as functions that convert integer lists into probability distributions for the next token. It thoroughly explains the tokenization process, which transforms raw text into numerical sequences, highlighting its critical role in LLM behavior and common failure modes like character-counting errors. The discussion covers Unicode, byte encoding (UTF-8), and the Byte Pair Encoding (BPE) algorithm, contrasting character-level with the prevalent byte-level BPE used in models like GPT-2, GPT-4, and Llama. It further distinguishes between tokenizer libraries such as SentencePiece and tiktoken, noting that both implement BPE but differ in speed and specific design choices. A comparative table illustrates the evolution of tokenizers across major models, revealing a trend towards larger vocabularies (e.g., GPT-4's 100,277, Llama 3's 128,000, GPT-4o's ~200,019, Gemma's 256,000-262,000 tokens) to enhance multilingual efficiency and reduce inference costs.

Key takeaway

For AI Engineers debugging unexpected LLM behavior, understanding the underlying tokenization process is crucial. You should recognize that LLMs operate on subword tokens, not individual characters, which explains failures in tasks requiring character-level awareness. When selecting models, consider the tokenizer's vocabulary size and type (e.g., byte-level BPE) for optimal multilingual performance and cost efficiency, especially for non-English applications. This knowledge allows you to anticipate model limitations and choose appropriate tools.

Key insights

LLMs operate on subword tokens, not characters, making tokenization a core determinant of their capabilities and predictable failure points.

Principles

• LLMs are functions predicting the next token's probability.
• Byte-level BPE tokenization prevents unknown tokens.
• Larger vocabularies improve multilingual efficiency and cost.

Method

The BPE algorithm starts with base units, iteratively merges the most frequent adjacent pairs into new units, and repeats until a target vocabulary size is reached, saving the merge order for inference.

In practice

• Analyze tokenization to predict LLM task failures.
• Use tiktoken for faster text-to-token conversion.
• Consider model's tokenizer for multilingual efficiency.

Topics

• Large Language Models
• Tokenization
• Byte Pair Encoding
• SentencePiece
• tiktoken
• Multilingual NLP

Best for: AI Engineer, Machine Learning Engineer, AI Student

Related on AIssential

• How Tokenization and Embeddings Actually Works In LLMs | Deep Dive — Tokenization and embeddings convert natural language into numerical representations, with positional embeddings preserving word order for LLMs.
• The Language of AI: Understanding Tokens in NLP — Tokens are the fundamental units AI uses to process text, impacting cost, memory, and understanding in LLMs.
• Why LLMs Get Math Wrong — Large Language Models (LLMs) often make mistakes in complex math problems and misinterpret tables because they process all input as flat text token sequences, treating math as patterns to predict rath…
• The Essence of LLM: Function — An LLM is a mathematical function mapping token sequences to probability distributions, driving all its emergent behaviors.
• Why Every Powerful LLM Can’t Spell “Strawberry” — And How Meta’s Byte Latent Transformer Finally… — LLMs struggle with character-level tasks due to tokenization, a problem Meta's BLT aims to solve by processing text without an alphabet.
• Building a Small Language Model (1) — Understanding Transformer — LLMs predict the next token by processing numerical representations of text through a multi-stage Transformer architecture.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MLWhiz: Recs|ML|GenAI.