Efficient Elicitation of Collective Disagreements

2026-05-21 · Source: cs.AI updates on arXiv.org · Field: Science & Research — Mathematics & Computational Sciences, Social Sciences & Behavioral Studies, Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

This research introduces the plurality matrix, a novel framework for analyzing voter disagreement by organizing aggregated preference data based on subset sizes. It defines the "level" of a disagreement measure as the smallest subset size needed to express it. The study demonstrates that many existing disagreement notions, including rank-variance and divisiveness, are level 3 measures, proving that simple pairwise comparisons (level 2) are insufficient to capture their full complexity. The hierarchy of information levels is shown to be strict, with k-central moments requiring level k+1 data. However, under specific structural assumptions like the Plackett-Luce model or single-peaked preferences, this hierarchy collapses to level 2. To make these findings actionable, the authors propose two elicitation protocols, k-chain and k-ranking, which explore the trade-off between per-voter cognitive load (e.g., k-1 for k-chain, k*log(k) for k-ranking) and the number of required participants for accurate estimation of the plurality matrix entries.

Key takeaway

For research scientists designing preference elicitation systems, recognize that standard pairwise comparisons often miss crucial disagreement structures. If your analysis involves measures like rank variance or divisiveness, you must collect higher-degree preference data, such as from subsets of three or more alternatives. Consider the trade-off between voter cognitive load and population size when choosing between k-chain or k-ranking protocols. This ensures your models accurately capture nuanced collective disagreements, avoiding misinterpretations of consensus.

Key insights

Pairwise comparisons are insufficient to capture complex voter disagreement; higher-order preference data is often required.

Principles

• Disagreement measures have inherent "levels" of information required.
• Plurality matrix generalizes pairwise comparisons to subsets.
• Structural preference models can simplify data requirements.

Method

The plurality matrix captures top-ranked probabilities for all alternative subsets. Elicitation protocols like k-chain and k-ranking estimate these, balancing voter cognitive load (e.g., k-1 vs. k*log(k)) against participant numbers for accuracy.

In practice

• Employ k-rankings for smaller voter pools, accepting higher cognitive load.
• Utilize k-chains for large populations, minimizing per-voter cognitive load.
• Check for Plackett-Luce or single-peaked preferences to simplify data needs.

Topics

• Plurality Matrix
• Disagreement Measures
• Preference Elicitation
• Social Choice Theory
• Cognitive Load
• Rank Variance

Best for: AI Scientist, Research Scientist

Related on AIssential

• Efficient Elicitation of Collective Disagreements — Pairwise comparisons are insufficient for capturing structural disagreement; a generalized approach is needed.
• Quantifying and Predicting Disagreement in Graded Human Ratings — Annotation disagreement in graded human ratings can be predicted from textual features, especially for inappropriate language.
• Survey Statistics: irrelevant alternatives ? — Multinomial logit models imply and are implied by the Independence from Irrelevant Alternatives (IIA) principle.
• Probably Approximately Consensus: On the Learning Theory of Finding Common Ground — Consensus can be modeled as an opinion interval, incorporating topic salience for effective online deliberation.
• Survey Statistics: Individualism and the CV Noise Problem — Individual-level log loss often fails to differentiate substantively important model improvements in binary data.
• Diversity of Extensions in Abstract Argumentation — A new quantitative measure, "diversity of extensions," quantifies the dissimilarity of accepted viewpoints in abstract argumentation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.