Remembering David MacKay

2026-03-01 · Source: Statistical Modeling, Causal Inference, and Social Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, short

Summary

The Cambridge Philosophical Society is organizing a meeting on "Energy and Information" to commemorate the 10th anniversary of David MacKay's death. The author shares a tribute focusing on MacKay's "Twelve Samples" anecdote from his 2003 information theory book. MacKay stated that "twelve independent samples" are often sufficient to estimate a parameter Φ to a precision finer than σ/3, specifically σ/√12. The author's confusion about the number "12" led to MacKay's humorous explanation that it originated from using "a dozen" in an earlier draft, later changed to "12" for clarity. The author also reflects on a disagreement with MacKay regarding Occam's razor, noting MacKay's characteristic generosity and commitment, contrasting him with less ethical figures in science. The post includes links to the conference and a form for sharing memories of MacKay.

Key takeaway

For researchers and practitioners working with statistical estimation, MacKay's "Twelve Samples" rule offers a practical guideline for data collection. If your goal is to estimate a parameter Φ to a precision of approximately σ/3, gathering around 12 independent samples can be a sufficient and efficient approach. This anecdote also highlights the value of clear communication over elegant language in technical writing.

Key insights

David MacKay's "Twelve Samples" rule of thumb for statistical precision stemmed from a "dozen" in an earlier draft.

Principles

• Precision of σ/√N is achievable with N samples.
• A precision of σ/3 is often sufficient for practical purposes.

In practice

• Consider N=12 samples for parameter estimation.
• Aim for precision around σ/3 in many estimation tasks.

Topics

• David MacKay
• Information Theory
• Statistical Sampling
• Occam's Razor
• Scientific Ethics

Best for: AI Researcher, Research Scientist, General Interest

Related on AIssential

• Optimal Deterministic Multicalibration and Omniprediction — Deterministic multicalibration can achieve minimax-optimal sample complexity, resolving a key open problem in trustworthy machine learning.
• Importance Sampling - Why Where You Sample Beats How Often — Importance Sampling efficiently estimates rare events by sampling from a targeted distribution and weighting samples.
• Most of Your Data is Useless — Sufficient statistics enable complete data compression for parameter estimation without any information loss.
• Variance Computation for Weighted Model Counting with Knowledge Compilation Approach — Computing WMC variance is tractable for structured d-DNNFs but intractable for broader DNNF variants.
• CALIBER: Calibrating Confidence Before and After Reasoning in Language Models — Confidence in reasoning language models is state-dependent, requiring distinct supervision targets for pre- and post-reasoning estimates.
• LOTTERY: Learning from Reference-Only Samples in Two-Sample Testing under Size Asymmetry — LOTTERY enables robust two-sample testing in few-shot, imbalanced settings by learning reference-dependent representations from abundant reference data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Statistical Modeling, Causal Inference, and Social Science.