Reinventing Entropy | Compression & Intelligence Part 1

2026-06-07 · Source: 3Blue1Brown · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Intermediate, extended

Summary

"Reinventing Entropy | Compression & Intelligence Part 1" delves into the fundamental limits of data compression, rooted in Claude Shannon's 1940s information theory. It establishes the mathematical equivalence between prediction and compression, reframing large language model pre-training as creating efficient text compressors rather than just next-token prediction. The article introduces core concepts like information and entropy using a robot instruction example with non-uniform probabilities (50% up, 25% down, 12.5% left, 12.5% right). It compares straightforward (2 bits/instruction) and clever (prefix-free, 1.75 bits/instruction) encoding methods, leading to the theoretical idea that perfect compression yields random noise. This insight defines information as -log₂(p) and entropy as the average information per symbol, H = Σ p * -log₂(p), representing the minimum bits per symbol for encoding. Shannon's early experiments, including human guessers, estimated English language entropy at about one bit per character with 100 preceding letters of context, setting up future discussions on cross-entropy in LLMs.

Key takeaway

For Machine Learning Engineers optimizing model pre-training, understanding the deep connection between prediction and compression is crucial. Your cross-entropy loss objective directly relates to building the most efficient data compressor, not just predicting the next token. This perspective can clarify why certain training strategies are effective and guide your approach to model architecture and data efficiency, especially when considering the theoretical limits of language compressibility.

Key insights

Prediction and compression are mathematically equivalent, forming the basis for information theory and LLM pre-training objectives.

Principles

• More common data should use fewer bits for efficiency.
• Perfect compression algorithms produce random noise.
• No code word can be a prefix of another code word.

Method

A prefix-free code assigns shorter bit strings to more probable symbols, ensuring unambiguous decoding by reading the bitstream until a complete code word forms.

In practice

• Use prefix-free codes for variable-length symbol encoding.
• Calculate average bits per symbol via a weighted sum of probabilities.

Topics

• Information Theory
• Data Compression
• Shannon Entropy
• Large Language Models
• Cross-Entropy Loss
• Prefix-Free Codes

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Entropy - Explained — Surprise, entropy, cross-entropy, and KL divergence quantify information, average uncertainty, encoding efficiency, and model fit respectively.
• Yes it's just doing compression. No it's not the diss you think it is. — Optimizing language models for next-word prediction via log-likelihood is a form of data compression, which can be linked to understanding.
• An Information-Theoretic Criterion for Efficient Data Synthesis — Synthetic data is effective only when external signals keep the training loop "information-open."
• Neural Network Compression by Approximate Differential Equivalence — Compress neural networks by aggregating functionally similar neurons using differential equivalence, not just pruning weights.
• Residual Modeling for High-Fidelity Learned Compression of Scientific Data — Tailored residual representations preserve learned compression advantages in high-fidelity scientific data scenarios.
• Towards Accurate and Calibrated Classification: Regularizing Cross-Entropy From A Generative Perspective — Generative Cross-Entropy improves DNN accuracy and calibration by regularizing class-level confidence from a generative perspective.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by 3Blue1Brown.