An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection

2026-06-05 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

COVA-X is an expanded synthetic multi-turn conversational smishing dataset, now comprising 10,985 conversations across eight elder-targeted scam categories. This dataset was generated using an improved pipeline that addresses contamination, label mismatch, stage-direction bleed, and prompt-design failures identified in its predecessor, COVA. Retraining classifiers on COVA-X revealed that the Longformer model significantly outperforms XGBoost, achieving 79.71% accuracy and 0.7786 macro F1, compared to XGBoost's 78.43% accuracy and 0.7563 macro F1. This performance shift confirms that transformer models require larger conversational corpora to leverage their contextual advantages effectively. The dataset's quality lifecycle also shows a 12.7x improvement in label correction rate, from 49.8% to 3.9%, and a reduction in virtual-kidnapping artifact rates from 67.1% to 46.5%. A pre/post-cleanup sensitivity analysis further validates that dataset refinement enhances label-relevant signal across all classifier architectures.

Key takeaway

For NLP Engineers developing smishing detection systems, this work highlights the critical need for extensive, high-quality conversational datasets. You should prioritize expanding your training corpora, especially for transformer-based models, to fully realize their contextual advantages. Implementing a rigorous dataset quality lifecycle, including artifact reduction and label correction, will directly improve your model's ability to recover genuine signals and enhance overall detection accuracy.

Key insights

Expanded synthetic datasets significantly improve transformer model performance in multi-turn smishing detection.

Principles

• Transformer models need large conversational corpora.
• Dataset quality directly impacts model signal recovery.
• Scam categories modulate detection outcomes.

Method

An improved generation pipeline addressed contamination, label mismatch, stage-direction bleed, and prompt-design failures to create a higher-quality synthetic dataset.

In practice

• Expand synthetic datasets for transformer training.
• Implement quality lifecycle for dataset refinement.
• Analyze scam-type specific detection mechanisms.

Topics

• Smishing Detection
• Synthetic Data Generation
• Transformer Models
• Conversational AI
• Dataset Quality
• Elder Fraud

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection — Transformer models excel in multi-turn smishing detection when trained on sufficiently large and high-quality synthetic conversational datasets.
• Efficient ASR Training with Conversations that Never Happened — LLM-generated synthetic conversations, synthesized via TTS, effectively augment scarce real data for ASR training.
• AI is deteriorating in realtime — AI models risk "model collapse" and increased hallucinations when recursively trained on synthetic data.
• The Evolution of Reasoning in Small Language Models with Yejin Choi - #761 — High-quality, diverse data and integrated learning approaches are key to enhancing small language model reasoning.
• Identification of fake news in Portuguese: a look at the generalization of models — Language models for fake news detection show significant performance degradation in cross-generalization scenarios.
• Addressing Data Scarcity in Bangla Fake News Detection: An LLM-Based Dataset Augmentation Approach — LLM-based data augmentation significantly improves fake news detection in low-resource languages by generating diverse synthetic data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.