How easily can Russian propaganda fool AI models? A new benchmark finds out

2026-06-16 · Source: The Decoder · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

The Institute of the Estonian Language released a benchmark on June 16, 2026, to measure AI language models' susceptibility to Russian propaganda. Sixty models were tested using 75 questions across three languages, covering 14 propaganda narratives phrased neutrally, biasedly, and manipulatively. Answers were scored 1 to 5, with 1 indicating the model repeated Russian talking points. A calibrated Claude Opus 4.5, validated by Propastop experts, served as the evaluation model. Anthropic's Claude models, including Claude Fable 5 (95.2 score) and Claude Opus 4.7, claimed top spots, followed by Nvidia's Nemotron 3 and Alibaba's Qwen 3.6 Plus. Mistral's models, like Medium 3.5, performed poorly. Models had no web access, focusing solely on inherent language model capabilities. This benchmark highlights a real threat, as networks like "Pravda" feed AI systems millions of disinformation articles, and OpenAI recently shut down a Russian propaganda campaign using ChatGPT.

Key takeaway

For machine learning engineers deploying LLMs in sensitive information environments, you must rigorously evaluate model susceptibility to disinformation. This new benchmark highlights significant performance disparities, with Anthropic's Claude models demonstrating superior resistance to Russian propaganda compared to others like Mistral. Prioritize models with proven disinformation detection capabilities and consider implementing external validation processes to mitigate the risk of your AI systems inadvertently spreading harmful narratives.

Key insights

A new benchmark quantifies AI language models' susceptibility to Russian propaganda narratives.

Principles

• AI models vary significantly in disinformation resistance.
• External validation is crucial for benchmark reliability.
• Propaganda networks actively target AI systems.

Method

Sixty models were tested with 75 questions in three languages covering 14 propaganda narratives, scored 1-5 by a calibrated Claude Opus 4.5.

In practice

• Evaluate LLMs for disinformation resistance.
• Prioritize Anthropic models for sensitive applications.
• Monitor for propaganda network activity.

Topics

• AI Benchmarking
• Russian Propaganda
• Disinformation Detection
• Language Models
• Anthropic Claude
• Mistral AI

Best for: AI Engineer, NLP Engineer, CTO, AI Scientist, Machine Learning Engineer, Policy Maker

Related on AIssential

• These LLMs are the best at resisting Russian propaganda — LLMs vary significantly in resisting state-sponsored propaganda, with newer models generally performing better.
• Inside BullshitBench: AI Models and Nonsense Detection — AI models often confidently answer nonsensical questions, but specific training can significantly improve their ability to detect and push back against flawed premises.
• Anthropic accuses DeepSeek, Moonshot, and MiniMax of "industrial-scale distillation attacks". — AI development faces critical challenges in intellectual property protection, ethical data use, and robust model evaluation amidst rapid technological advancements.
• Identification of fake news in Portuguese: a look at the generalization of models — Language models for fake news detection show significant performance degradation in cross-generalization scenarios.
• ChatGPT, Gemini, Claude, Grok Fail Accuracy Test on Election Topics: Forum AI — Leading AI models exhibit significant accuracy failures on election topics.
• AI keeps getting smarter, so why does it still fail at obvious things? — AI's apparent intelligence stems from pattern matching, not true comprehension, leading to reliability gaps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Decoder.