The Kappa Zoo: David Eubanks’s online monograph on rating models

2026-05-28 · Source: Statistical Modeling, Causal Inference, and Social Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

David Eubanks's online monograph, "The Kappa Zoo," provides a comprehensive overview of rating and crowdsourcing models, despite being labeled a work in progress. The monograph details Bayesian rating models, tracing their origins to Phil Dawid and Allan Skene's 1979 paper. It explores extensive workflow considerations and various model evaluation and comparison measures, connecting them to information-theoretic concepts like entropy. A significant section critically examines Cohen's kappa statistic, arguing it fails to adequately measure inter-rater agreement, a conclusion consistent with other research. Furthermore, Eubanks includes a valuable comparison of Item-Response Theory (IRT) models incorporating difficulty parameters, a topic highlighted as crucial for advancing crowdsourcing methodologies, aligning with recent work on arXiv:2405.19521.

Key takeaway

For research scientists or data scientists designing crowdsourcing systems or evaluating human-generated ratings, you should consult David Eubanks's "The Kappa Zoo." This resource provides a critical perspective on common metrics like Cohen's kappa and highlights the importance of advanced Item-Response Theory (IRT) models that account for item difficulty. Incorporating these insights can significantly improve the accuracy and reliability of your rating model evaluations and crowdsourcing task designs.

Key insights

The Kappa Zoo monograph offers a critical overview of rating models, evaluating methods from Dawid and Skene to IRT with difficulty.

Principles

• Inter-rater agreement metrics require careful scrutiny.
• IRT models can incorporate item difficulty for better ratings.
• Bayesian workflow is key for model comparison.

Method

The monograph implicitly outlines a workflow for evaluating rating models, using information-theoretic measures and comparing IRT models with and without difficulty parameters.

In practice

• Consult "The Kappa Zoo" for rating model selection.
• Re-evaluate Cohen's kappa for agreement tasks.
• Explore IRT models for crowdsourcing with item difficulty.

Topics

• Rating Models
• Crowdsourcing
• Bayesian Statistics
• Item-Response Theory
• Model Evaluation
• Inter-rater Agreement

Best for: AI Scientist, Data Scientist, Research Scientist

Related on AIssential

• Quantifying and Predicting Disagreement in Graded Human Ratings — Annotation disagreement in graded human ratings can be predicted from textual features, especially for inappropriate language.
• The Bayesian Workflow book is coming! — The "Bayesian Workflow" book offers practical guidance for applying Bayesian methods, addressing common challenges in statistical and computational practice.
• An Interpretable and Scalable Framework for Evaluating Large Language Models — cBMM offers a scalable, interpretable framework for LLM evaluation by modeling latent abilities and item characteristics.
• Applied Item Response Theory (1968 psychometrics) to 242K cancer drug sensitivity measurements — IRT recovers rankings where averaging fails under sparsity — Item Response Theory effectively ranks cancer drug sensitivity from sparse data where averaging fails.
• Building better AI benchmarks: How many raters are enough? — Optimizing rater-to-item ratios is crucial for reproducible AI benchmarks that capture human disagreement.
• The social sciences need tools for the 21st century | Letters — Social science needs new tools and methods to improve replicability and advance understanding of human behavior.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Statistical Modeling, Causal Inference, and Social Science.