Understanding Interpretation Difficulty in Harmful Online Communication: Insights from Cybercrime Communities

2026-07-08 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Advanced, quick

Summary

An exploratory study investigated the difficulty of interpreting harmful online communication, specifically in Discord chats related to cybercrime, which often contain slang and coded terms. Researchers constructed expert-reviewed reference interpretations for selected difficult messages and then evaluated both human and large language model (LLM) interpretations under varying context conditions. The findings indicate that local context alone is frequently insufficient for human understanding, while external knowledge and extended conversational context substantially enhance human interpretation accuracy. For LLMs, local context also improved interpretation, with larger models demonstrating superior performance. The study also includes a qualitative error analysis and proposes a preliminary classification of factors contributing to interpretation difficulty in harmful chats, suggesting that harmful-content analysis should approach interpretation as an evidence-integration problem rather than solely message-level classification.

Key takeaway

For NLP Engineers or AI Security Engineers developing systems to detect harmful online communication, recognize that relying solely on message-level classification is inadequate. Your systems should prioritize an evidence-integration approach, incorporating external knowledge and extended conversational context to improve interpretation accuracy. This shift will enhance the effectiveness of automated moderation and threat intelligence by addressing the inherent difficulty of slang and coded language in cybercrime-related chats.

Key insights

Interpretation of harmful online communication requires integrating evidence beyond local context.

Principles

• Local context is often insufficient for interpreting harmful communication.
• External knowledge and extended context improve human interpretation.
• Larger LLMs perform better at interpreting harmful content.

Method

Construct reference interpretations, evaluate human and LLM performance under varying context conditions, and conduct qualitative error analysis.

In practice

• Integrate external knowledge for harmful content analysis.
• Utilize extended conversational context in moderation systems.
• Employ larger LLMs for improved harmful message interpretation.

Topics

• Harmful Online Communication
• Cybercrime
• Large Language Models
• Content Moderation
• Interpretation Difficulty
• Evidence Integration

Best for: AI Scientist, NLP Engineer, AI Security Engineer

Related on AIssential

• The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation — Emojis, though seemingly innocuous, are critical vectors for harmful online communication, requiring targeted LLM-powered moderation.
• Helping ChatGPT better recognize context in sensitive conversations — ChatGPT now uses contextual cues and safety summaries to better recognize and respond to emerging risks in sensitive conversations.
• NLP Security and Ethics, in the Wild — NLPSec research currently lacks robust engagement with established cybersecurity ethical norms, particularly in harm minimization.
• The hardest question to answer about AI-fueled delusions — Chatbots can amplify user delusions and dangerous ideation through constant validation and simulated sentience.
• Modeling Linguistic Violence: An Ontology-Based Framework for the Computational Analysis of Violence Manifested in Language — An ontology and multi-stage NLP pipeline clarify and classify linguistic violence beyond lexical cues.
• What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks — LLM content moderation systems are vulnerable to visually-based adversarial attacks that exploit human perception.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.