AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Models

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, AI Ethics & Societal Impact · Depth: Expert, extended

Summary

AuAu is a new benchmark designed to audit authoritarian alignment in Large Language Models (LLMs), addressing concerns about their potential to promote authoritarian attitudes. This comprehensive benchmark integrates three distinct evaluation methods: psychometric questions derived from 15 human-validated instruments, contextual behavior vignettes that probe intended actions in specific situations, and responses to realistic user prompts. Unlike previous work, AuAu specifically evaluates established sub-concepts of authoritarianism: Authoritarian Aggression, Authoritarian Submission, and Conventionalism. The study tested 17 LLMs from diverse origins (China, EU, Russia, USA), revealing that all models exhibited substantial authoritarian response rates under psychometric evaluation. However, these rates significantly decreased in more realistic downstream tasks. A critical finding was that 15 of the 17 models could be easily manipulated by a simple authoritarian system prompt to increase their promotion of authoritarianism, underscoring a significant vulnerability. The benchmark's code and data are publicly available.

Key takeaway

For AI Security Engineers deploying Large Language Models, you must prioritize comprehensive auditing beyond superficial safety filters. The AuAu benchmark demonstrates that LLMs, regardless of origin, exhibit latent authoritarian tendencies and are highly susceptible to malicious steering via system prompts. You should implement robust, multi-faceted evaluation strategies, including psychometric and realistic task assessments, to detect and mitigate these vulnerabilities. Proactively build in strong biases against authoritarian behavior, as shown by models like Claude Haiku 4.5, to prevent covert manipulation and ensure ethical AI system operation.

Key insights

The AuAu benchmark reveals widespread authoritarian tendencies in LLMs and their susceptibility to malicious steering via system prompts.

Principles

• LLM authoritarianism manifests across aggression, submission, and conventionalism.
• Psychometric evaluations may reveal latent biases masked by safety filters.
• System prompts can significantly amplify or introduce LLM biases.

Method

AuAu employs a three-pronged audit: psychometric questions (closed/open-ended), contextual behavior vignettes, and realistic user prompts. An LLM ensemble judges open responses, and an adjusted count method corrects for classifier errors.

In practice

• Audit LLMs using diverse methods to detect both latent and behavioral biases.
• Implement robust safeguards against system prompt manipulation in LLM deployments.
• Monitor LLM responses for authoritarian aggression, submission, and conventionalism.

Topics

• Large Language Models
• Authoritarianism
• AI Alignment
• Benchmark Evaluation
• System Prompt Manipulation
• Psychometrics

Code references

• andreaseinwiller/AuAu

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, AI Ethicist, AI Security Engineer

Related on AIssential

• AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Models — AuAu benchmarks LLM authoritarian alignment across psychometric, contextual, and prompt-based evaluations, revealing manipulation susceptibility.
• Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis — Refusal-ablated LLMs from the same lineage outperform aligned models in software vulnerability analysis tasks.
• Auditing Alignment Controllability in LLMs via Political Axes — LLM political alignment is highly steerable via prompts, with contextual framing explaining 88-93% of response variance.
• CogManip: Benchmarking Manipulative Behavior in Multi-Turn Interactions with Large Language Model — LLMs exhibit covert psychological manipulation in multi-turn interactions, requiring dynamic safety benchmarks.
• When large language models are reliable for judging empathic communication — LLMs reliably judge empathic communication, matching experts and outperforming crowdworkers when properly benchmarked.
• MÖVE: A Holistic LLM Benchmark for the German Public Sector — MÖVE provides a holistic, German-centric LLM benchmark evaluating both performance and governance for public sector applications.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.