CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

2026-04-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

A new benchmark called CoPA has been introduced to evaluate personalized Question Answering (QA) capabilities of Large Language Models (LLMs). This benchmark addresses the limitations of existing evaluation paradigms, which often rely on lexical similarity or manual heuristics without sufficient data-driven validation. CoPA distills six key personalization factors by mining Community-Individual Preference Divergence (CIPD), where individual choices override consensus. It includes 1,985 user profiles for fine-grained, factor-level assessment. By quantifying the alignment between model outputs and user-specific cognitive preferences inferred from interaction patterns, CoPA offers a more comprehensive and discriminative standard for evaluating personalized QA compared to generic metrics. The code for CoPA is publicly available on GitHub.

Key takeaway

For research scientists developing or deploying LLMs for personalized Question Answering, CoPA provides a robust, data-driven benchmark to assess model performance beyond generic metrics. You should integrate CoPA into your evaluation pipeline to gain fine-grained, factor-level insights into how well your models align with individual user cognitive preferences, thereby improving personalization accuracy.

Key insights

CoPA benchmarks personalized QA by aligning LLM outputs with user-specific cognitive preferences derived from individual data.

Principles

• Individual preferences often diverge from community consensus.
• Personalization requires data-driven cognitive factor evaluation.

Method

CoPA mines Community-Individual Preference Divergence (CIPD) to distill six personalization factors. It then quantifies alignment between model outputs and user-specific cognitive preferences inferred from interaction patterns.

In practice

• Use CoPA to evaluate LLMs for personalized QA.
• Analyze CIPD to identify user-specific preferences.

Topics

• Personalized Question Answering
• CoPA Benchmark
• Community-Individual Preference Divergence
• Cognitive Factors
• Large Language Models

Code references

• bjzgcai/CoPA

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Causal_QA.PT: A Human–LLM Co-Curated Benchmark for Causal Question Answering in Portuguese Language — A new human-LLM co-curated benchmark improves causal reasoning evaluation for Portuguese LLMs.
• MARCA: A Checklist-Based Benchmark for Multilingual Web Search — MARCA evaluates LLM web search reliability using bilingual questions and checklist rubrics.
• DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering — DiscoTrace reveals LLMs lack rhetorical diversity and over-address question interpretations compared to human answerers.
• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions — LLMs achieve high semantic alignment through paraphrasing, not lexical reproduction, challenging traditional evaluation metrics.
• Human-Centric Topic Modeling with Goal-Prompted Contrastive Learning and Optimal Transport — Integrating human-provided goals directly into topic modeling improves topic interpretability, diversity, and goal alignment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.