PSK@EEUCA 2026: Fine-tuning Large Language Models with Synthetic Data Augmentation for Multi-class Toxicity Detection in Gaming Chat

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Srikar Kashyap Pulipaka's system for the EEUCA 2026 Shared Task addresses multi-class toxicity detection in World of Tanks gaming chat. The system classifies messages into six categories: Non-toxic, Insults/Flaming, Other Offensive, Hate/Harassment, Threats, and Extremism. The research explored various methods, including encoder-based models, instruction-tuned LLMs with LoRA fine-tuning, and ensemble techniques. Their top-performing system utilized Llama 3.1 8B, enhanced with a carefully calibrated 5% synthetic data augmentation. This approach achieved an F1-macro score of 0.6234 on the test set, securing 4th place among 35 participating teams. The analysis also identified a "validation trap," where strong validation performance did not translate to effective test set generalization, linked to dataset annotation patterns.

Key takeaway

For machine learning engineers developing toxicity detection systems, you should critically evaluate dataset annotation patterns to avoid the "validation trap." Your models, even when fine-tuned with synthetic data like Llama 3.1 8B, may show high validation scores but fail to generalize on unseen test data. Prioritize robust test set performance over validation metrics, and carefully calibrate synthetic data augmentation to ensure real-world effectiveness in gaming chat moderation.

Key insights

Synthetic data augmentation and Llama 3.1 8B can effectively detect multi-class toxicity, but a "validation trap" can hinder test set generalization.

Principles

• Careful synthetic data calibration improves LLM performance.
• Dataset annotation patterns impact model generalization.
• High validation F1-macro does not guarantee test transfer.

Method

The system fine-tuned Llama 3.1 8B using LoRA and 5% synthetic data augmentation, exploring hierarchical, one-vs-rest, and ensemble strategies for multi-class toxicity detection.

In practice

• Apply Llama 3.1 8B with LoRA for chat moderation.
• Calibrate synthetic data augmentation carefully.
• Analyze dataset annotation for generalization issues.

Topics

• Large Language Models
• Toxicity Detection
• Synthetic Data Augmentation
• Llama 3.1
• LoRA Fine-tuning
• Gaming Chat Moderation
• Validation Trap

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Addressing Data Scarcity in Bangla Fake News Detection: An LLM-Based Dataset Augmentation Approach — LLM-based data augmentation significantly improves fake news detection in low-resource languages by generating diverse synthetic data.
• Detoxification for LLM: From Dataset Itself — Detoxifying LLM pretraining datasets directly at the source fundamentally reduces inherent model toxicity while preserving semantics.
• I Fine-Tuned an LLM to Reply Like My Wife — 10 Years of Chats, One Adapter, and Results That Amazed… — Fine-tuning LLMs with personal, messy data on consumer hardware can yield surprising persona learning.
• Designing a Synthetic Data Pipeline for Persian LLM Fine Tuning — Data quality and engineering are paramount for LLM performance, especially in low-resource languages.
• DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning — DataShield efficiently identifies safety-degrading samples in benign LLM fine-tuning datasets by quantifying compliance behavior.
• The threat of analytic flexibility in using large language models to simulate human data — Analytic choices in LLM-generated synthetic data significantly alter its fidelity to human responses.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.