Building Multi-turn Intent Classification with LLM-based Labeling
Summary
A scalable multi-turn intent classification framework is proposed for e-commerce customer service, addressing challenges like complex intent taxonomies, dynamic intent switching, and limited labeled data. The framework utilizes LLM-based labeling strategies to annotate real customer transcripts at scale and augments training data with LLM-simulated multi-turn dialogues, which expand coverage of rare topic and intent switches. Experiments reveal that explanation-guided labeling, incorporating a self-critique step, yields the most accurate training labels. Fine-tuned models, built on a RoBERTa backbone, demonstrate superior performance over zero-shot LLM prompting, alongside significantly lower inference latency. Furthermore, a hybrid approach combining the fine-tuned classifier with LLM prompting achieves even higher accuracy than either component alone, offering practical guidance for deploying high-accuracy, low-latency, large-scale systems.
Key takeaway
For Machine Learning Engineers developing multi-turn intent classification systems in e-commerce, you should integrate LLM-based labeling with self-critique to overcome data scarcity and complex taxonomies. Augment your training data with LLM-simulated dialogues to cover dynamic intent switches. Deploying a hybrid system, combining a fine-tuned RoBERTa classifier with LLM prompting, will achieve superior accuracy and lower inference latency compared to either approach alone, optimizing customer service routing.
Key insights
Combining LLM-based data labeling, simulated dialogues, and hybrid classification improves multi-turn intent accuracy and efficiency.
Principles
- Explanation-guided labeling with self-critique enhances label accuracy.
- Fine-tuned models offer lower latency than zero-shot LLMs.
- Hybrid classification outperforms individual components.
Method
Annotate real transcripts using explanation-guided LLM labeling with self-critique. Augment training with LLM-simulated multi-turn dialogues. Fine-tune a RoBERTa-based classifier and combine with LLM prompting for a hybrid system.
In practice
- Use LLMs for scalable, accurate data labeling.
- Generate synthetic dialogues for rare intent switches.
- Deploy hybrid RoBERTa + LLM systems for optimal results.
Topics
- Multi-turn Intent Classification
- LLM-based Labeling
- Data Augmentation
- RoBERTa
- Hybrid AI Systems
- Customer Service AI
Best for: AI Engineer, AI Architect, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.