When Scanners Lie: Evaluator Instability in LLM Red-Teaming

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

Automated LLM vulnerability scanners, used to assess security risks by measuring attack success rates (ASR), suffer from significant measurement instability due to their evaluator component. A study demonstrates that changing the evaluator, while keeping attacks and model outputs constant, can substantially alter reported ASRs. To address this, a two-phase, reliability-aware evaluation framework is introduced. The first phase quantifies evaluator disagreement to identify unreliable ASR categories. The second phase proposes a verification-based method where evaluators are validated by an independent verifier, enabling reliability assessment without extensive human annotation. Applied to the Garak scanner, 22 of 25 attack categories exhibited evaluator instability, with the framework raising evaluator accuracy from 72% to 89%. Reported vulnerability scores can vary by up to ±33% depending on the evaluator.

Key takeaway

For AI Security Engineers assessing LLM security risks, current automated vulnerability scanner outputs are highly sensitive to evaluator choice, leading to unreliable Attack Success Rate (ASR) measurements. You should implement a reliability-aware evaluation framework to quantify evaluator disagreement and validate evaluators. This approach enhances measurement accuracy from 72% to 89% and provides a practical method to control costs while ensuring more trustworthy security assessments.

Key insights

LLM vulnerability scanner reliability hinges on evaluator stability, which is often compromised, leading to significant ASR variations.

Principles

• Evaluator choice critically impacts LLM vulnerability assessment.
• Quantify evaluator disagreement to identify unreliable ASR categories.
• Independent verification enhances evaluator accuracy and reliability.

Method

A two-phase framework: first, quantify evaluator disagreement; second, use an independent verifier for validation, enabling reliability assessment without extensive human annotation.

In practice

• Apply a verification-based method to validate LLM evaluators.
• Identify attack categories with high evaluator disagreement.
• Improve evaluator accuracy from 72% to 89%.

Topics

• LLM Red-Teaming
• Vulnerability Scanners
• Evaluator Instability
• Attack Success Rate
• Reliability Assessment
• Garak Scanner

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Security Engineer, MLOps Engineer, AI Scientist

Related on AIssential

• AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability — AdversaBench automates LLM red-teaming by generating adversarial prompts and confirming failures with a multi-judge system.
• Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software — LLMs fine-tuned for vulnerability detection achieve "calibration without comprehension," adapting outputs without true security reasoning.
• How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation — LLM search agents exhibit varied endorsement vulnerability to web content manipulation, requiring robust safety evaluations.
• Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration — Abliteration separates code LLM refusal from capability, unlocking vulnerability injection without creating new skills.
• When Behavioral Safety Evaluation Fails: A Representation-Level Perspective — Current LLM behavioral safety evaluations are insufficient, failing to detect deep representation-level vulnerabilities that an "audit gap" reveals.
• Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering — LLM agents significantly reduce SAST false positives, but effectiveness is highly dependent on the backbone model and vulnerability category.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.