Adversarial Creation and Detection of AI-Generated Social Bot Content

2025-08-07 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

This paper introduces an adversarial methodology and a new multilingual, cross-platform dataset designed to improve the detection of AI-generated social bot content. The methodology models malicious actors impersonating real social media users, generating synthetic messages conditioned on user profiles and historical behaviors across 36 Reddit and Telegram channels in 17 languages. The resulting dataset contains 73,521 unique real user messages and 263,594 paired real and AI-generated texts. Models trained on this adversarial data, particularly transformer-based classifiers (TC), significantly outperform existing content-based bot detection baselines on real-world, out-of-distribution data like the Fox8-23 dataset. The best TC model achieved near-perfect accuracy for detecting AI-powered social bots at the user level, demonstrating the value of realistic, context-aware training data. Detection accuracy increases with message length and is harder for content from larger LLMs or when conversational context is provided.

Key takeaway

For AI Security Engineers or NLP Engineers building social media bot detection systems, you should prioritize training your models on adversarially generated, context-aware datasets. This approach, which mimics sophisticated bot behavior, yields significantly higher accuracy in identifying AI-powered social bots at the user level on real-world data. Continuously refresh your training data with outputs from state-of-the-art LLMs and varied prompting strategies to counter evolving bot capabilities.

Key insights

Training AI-generated content detectors on adversarially created, context-aware data significantly improves real-world bot detection accuracy.

Principles

• Realistic adversarial data improves bot detection.
• Contextual generation makes AI text harder to detect.
• User-level detection is more accurate than message-level.

Method

Curate real social media conversations, then use LLMs to generate adversarial messages imitating specific users based on their persona and conversational context. Train classifiers on this paired data.

In practice

• Develop detection models using adversarially generated data.
• Focus on user-level bot detection for higher accuracy.
• Continuously update training data with new LLM outputs.

Topics

• AI-Generated Content Detection
• Social Bots
• Adversarial Data Generation
• Large Language Models
• Social Media Security
• Multilingual NLP

Code references

• pemistahl/lingua-py
• brucewlee/lftk
• trokhymovych/ai-bot-detection

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Security Engineer

Related on AIssential

• Adversarial Creation and Detection of AI-Generated Social Bot Content — Adversarial training with paired human and AI-generated data significantly improves social bot content detection.
• A viral Reddit post alleging fraud from a food delivery app turned out to be AI-generated — AI tools enable sophisticated online hoaxes, challenging traditional fact-checking and content verification methods.
• Generative AI and Digital Ecosystem Resilience: A Proactive Lifecycle-Based Survey — Proactive, lifecycle-based detection of GenAI-driven inauthentic narratives is crucial for digital ecosystem resilience, shifting from reactive methods.
• Why Social Media Text Breaks AI Models — AI struggles with social media text due to its informality, code-mixing, context-dependency, and rapid evolution.
• Psychologically Potent, Computationally Invisible: LLMs Generate Social-Comparison Triggers They Fail to Detect — LLMs can generate social comparison triggers but struggle to reliably detect them via prompt-based classification.
• Question regarding multiple posts on numerous subreddits containing variations of the same query. — Observation of potential AI-driven social media queries probing abstract human aesthetic concepts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.