But what is cross-entropy? | Compression is Intelligence Part 2

2026-07-16 · Source: 3Blue1Brown · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, extended

Summary

The concept of cross-entropy, a fundamental measure in information theory, is explored through its surprising connection to data compression and its critical role in training modern language models. The 2002 paper "Language Trees and Zipping" demonstrated how general file compression tools like gzip could cluster languages and recover lineage trees by measuring co-compression efficiency. This principle, where a compression scheme optimized for one context performs in another, directly relates to cross-entropy. The article illustrates this with a robot instruction example, showing how an encoding optimized for one probability distribution (Q) performs on a new distribution (P), resulting in an average of 2.625 bits per message. In machine learning, cross-entropy loss, typically using natural logarithms for gradient descent, serves as the objective function for pre-training language models, minimizing when the model's predicted token distribution matches the training data's statistics. It also underpins distillation, where smaller models learn from larger ones' full probability distributions.

Key takeaway

For Machine Learning Engineers designing loss functions or optimizing model training, understanding cross-entropy's deep connection to compression is crucial. Your choice of a negative logarithm for loss is mathematically forced if you want the model's output distribution to match data statistics, ensuring optimal learning. This perspective reframes LLM training as a compression task, guiding you to consider distillation with cross-entropy for efficient model approximation, rather than just next-token prediction.

Key insights

Cross-entropy quantifies how well a compression scheme optimized for one context performs in another, linking compression to AI.

Principles

• Optimal code bits = -log2(probability).
• Cross-entropy is minimized when P equals Q.
• Logarithms are essential for loss function minimization.

Method

Language models are trained by minimizing cross-entropy loss, which is the average negative log probability the model assigns to true next tokens across vast datasets.

In practice

• Use co-compression for document similarity.
• Apply cross-entropy for author identification.
• Distill large models into smaller, efficient ones.

Topics

• Cross-Entropy
• Language Models
• Data Compression
• Loss Functions
• Model Distillation
• KL Divergence

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Reinventing Entropy | Compression & Intelligence Part 1 — Prediction and compression are mathematically equivalent, forming the basis for information theory and LLM pre-training objectives.
• Cross-Entropy is Surprise on the Wrong Distribution — Cross-entropy measures average surprise when reality (P) is scored by a model's belief (Q).
• Entropy - Explained — Surprise, entropy, cross-entropy, and KL divergence quantify information, average uncertainty, encoding efficiency, and model fit respectively.
• Loss Function Explained For Noobs (How Models Know They Are Wrong) — A loss function quantifies model prediction errors, guiding iterative adjustments to improve accuracy.
• How Does a Large Language Model Actually Learn? — LLMs acquire complex capabilities through iterative next-token prediction and error-driven weight adjustments.
• Logistic Regression (and why it's different from Linear Regression) — Logistic Regression uses a sigmoid function and cross-entropy loss for binary classification probability predictions.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by 3Blue1Brown.