The Poisoned Chalice of LLM Evaluation Report

2026-07-09 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cybersecurity & Data Privacy · Depth: Advanced, long

Summary

The Poisoned Chalice of LLM Evaluation Competition, co-located with FSE-AIWare 2026, addresses the critical issue of data contamination in large language model (LLM) evaluation for software engineering. This problem arises when LLMs perform well on benchmarks due to prior exposure to training data, rather than genuine generalization, especially problematic for models with undisclosed training corpora. The competition framed this as a white-box membership inference task on source code, providing participants with curated datasets from The Stack Edu and The Heap, targeting StarCoder2 3B/7B models, and using Mellum-4B as a held-out evaluation model. Submissions were evaluated using AUC-ROC on unseen data to ensure generalizability. SERSEM, a structure-aware method, emerged as the clear winner, achieving AUC-ROC scores of 0.773 on StarCoder2-3B and 0.753 on Mellum, significantly surpassing other methods and baselines like Loss, MinK%Prob, and PAC. The competition highlights the challenge of robust contamination detection and the need for more efficient, generalizable methods.

Key takeaway

For AI Security Engineers or ML Engineers evaluating code LLMs, you must account for data contamination risks that inflate performance metrics. Relying solely on reported benchmarks without verifying training data exposure can lead to misleading conclusions about model capabilities. Implement robust white-box membership inference techniques, like those explored in the Poisoned Chalice competition, to detect prior exposure. Prioritize methods demonstrating strong generalization across diverse codebases and models, such as SERSEM, to ensure your evaluations are trustworthy and reflect true model performance.

Key insights

Data contamination in LLM evaluation for software engineering requires robust, generalizable white-box membership inference methods.

Principles

• LLM benchmark performance can be inflated by data contamination.
• Membership inference can detect contamination without full training corpus access.
• Generalization across models and datasets is crucial for robust evaluation.

Method

The competition used white-box membership inference on source code, providing labeled member/non-member datasets (The Stack Edu, The Heap) and target models (StarCoder2, Mellum). Evaluation was via AUC-ROC on held-out data.

In practice

• Use structure-aware methods like SERSEM for contamination detection.
• Prioritize methods that generalize across diverse code languages.
• Evaluate MIAs on held-out models and datasets.

Topics

• LLM Evaluation
• Data Contamination
• Membership Inference Attacks
• Software Engineering
• Code Language Models
• StarCoder2
• SERSEM

Code references

• AISE-TUDelft/PoisonedChalice

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Did the Model See the Benchmark During Training? Detecting LLM Contamination — A lightweight method estimates LLM benchmark contamination by analyzing model behavior when training data is unknown.
• Yet another experiment proves it's too damn simple to poison large language models — LLMs with web search are vulnerable to simple, low-cost data poisoning via fabricated online sources.
• Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs — The Mask-Proof pipeline enables scalable, reproducible evaluation of LLM step-level mathematical reasoning using masked-step tasks and an LLM-based judge.
• An Empirical Study of Security Calibration in Large Language Models for Code — LLMs for code are systematically overconfident in security, especially in complex repository contexts, making their self-assessments unreliable.
• LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training — Data contamination in RL post-trained LLMs can be reliably detected by analyzing layer-wise representation deviations.
• From Wordle to Fibble5: Evaluating LLM Reasoning Under Escalating Deception — LLM reasoning robustness is severely degraded by even minimal deception, a factor poorly captured by standard benchmarks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.