Understanding Interpretation Difficulty in Harmful Online Communication: Insights from Cybercrime Communities

2026-06-03 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

An exploratory study analyzed interpretation difficulty in 100 cybercrime-related Discord chat messages, comparing human and large language model (LLM) performance under varying context conditions. Researchers constructed expert-reviewed reference interpretations to evaluate human annotators and two open-weight LLMs, GPT-OSS-20B and GPT-OSS-120B. Results showed that human interpretation significantly improved from 2.7 to 62.7 matches out of 100 when external knowledge and full channel history were provided, whereas local context alone offered minimal improvement (5.3 matches). LLMs, particularly GPT-OSS-120B, achieved 58 matches with local context, outperforming humans in comparable limited-context settings. The study also classified factors contributing to interpretation difficulty, suggesting that harmful-content analysis requires integrating diverse evidence beyond simple message classification.

Key takeaway

For AI Security Engineers or NLP Engineers developing harmful content detection systems, recognize that message-level classification and limited local context are inadequate. Your systems must integrate external knowledge, such as domain-specific slang and community conventions, alongside extended conversational history to accurately interpret nuanced or coded harmful communication. Relying solely on LLMs with local context may miss critical implicit meanings, necessitating a robust evidence-integration approach for reliable analysis.

Key insights

Interpreting harmful online communication requires integrating diverse knowledge and context, as local cues are often insufficient.

Principles

• Local context is often insufficient for interpreting harmful messages.
• External knowledge and extended context substantially improve interpretation.
• Interpretation is an evidence-integration problem, not message classification.

Method

Construct reference interpretations of difficult messages, evaluate human/LLM interpretations under varying context, then qualitatively analyze errors to classify difficulty factors.

In practice

• Integrate external knowledge and full chat history for analysis.
• Consider LLMs for initial interpretation with local context.
• Focus on evidence integration over simple keyword detection.

Topics

• Cybercrime Analysis
• Harmful Content Detection
• Large Language Models
• Discord Chat Analysis
• Contextual Interpretation
• NLP for Security

Best for: AI Scientist, NLP Engineer, AI Security Engineer

Related on AIssential

• Understanding Interpretation Difficulty in Harmful Online Communication: Insights from Cybercrime Communities — Interpretation of harmful online communication requires integrating evidence beyond local context.
• The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation — Emojis, though seemingly innocuous, are critical vectors for harmful online communication, requiring targeted LLM-powered moderation.
• Helping ChatGPT better recognize context in sensitive conversations — ChatGPT now uses contextual cues and safety summaries to better recognize and respond to emerging risks in sensitive conversations.
• Analysing Self-Harm Representations in Language Models: a Cross-Architecture Study — Self-harm content representations in LLMs localize to specific layers and vary across architectures.
• Mapping the Landscape of Unregulated eXplicit Contents on Reddit — Nuanced detection of explicit online content requires analyzing expressive signals beyond keywords.
• NLP Security and Ethics, in the Wild — NLPSec research currently lacks robust engagement with established cybersecurity ethical norms, particularly in harm minimization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.