ByteBreaker@DravidianLangTech 2026: XLM-RoBERTa Large with Sliding-Window Chunking and Top-K Mean Pooling for Writing Style Classification

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, medium

Summary

ByteBreaker, a system developed for the Prompt Recovery for LLM Shared Task at DravidianLangTech@ACL-2026, classifies writing styles in large language model (LLM) generated documents. It identifies nine styles: Authoritative, Formal, Humorous, Informal, Inspiring, Optimistic, Persuasive, Pessimistic, and Serious. To handle documents exceeding the 512 token limit of transformer models, ByteBreaker employs a sliding-window method, breaking text into overlapping 512-token chunks with a 256-token stride. The system fine-tunes XLM-RoBERTa Large using only "CHANGE STYLE" text for its distinct stylistic indicators. For prediction, it utilizes Top-K mean pooling to emphasize confident chunk-level predictions. After training with five random seeds, three submissions were made: a weighted ensemble (Run 1), a mean-guided single model (Run 2), and a Top-K-guided single model (Run 3). Run 3 achieved the highest macro F1 score of 0.3306, while Run 1 recorded the best accuracy of 0.3256 with a macro F1 of 0.3290.

Key takeaway

For NLP Engineers developing writing style classifiers for LLM outputs, consider implementing a sliding-window approach with Top-K mean pooling. This strategy effectively handles long documents and improves prediction confidence by emphasizing the most certain chunk-level classifications. You should fine-tune models like XLM-RoBERTa Large on stylistically distinct text, such as "CHANGE STYLE" examples, to achieve better performance. Experiment with ensemble methods or multiple random seeds to enhance the robustness and consistency of your classification system.

Key insights

Sliding-window chunking and Top-K mean pooling enhance transformer models for long-document writing style classification.

Principles

• Fine-tune on text with distinct stylistic indicators.
• Prioritize confident predictions in chunk-level aggregation.
• Ensemble models or use multiple seeds for consistency.

Method

Documents are chunked into overlapping 512-token segments with a 256-token stride. XLM-RoBERTa Large is fine-tuned, and Top-K mean pooling aggregates chunk predictions for final style classification.

In practice

• Implement sliding windows for long text processing.
• Apply Top-K mean pooling for robust prediction.
• Use "CHANGE STYLE" text for style-specific fine-tuning.

Topics

• Writing Style Classification
• LLM Prompt Recovery
• XLM-RoBERTa Large
• Sliding-Window Chunking
• Top-K Mean Pooling
• Transformer Models

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Chunking Strategies for Documents in RAG-Powered AI Summarizers — Effective document chunking is critical for RAG pipeline performance, directly impacting retrieval quality and LLM context.
• Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models — Kronecker Embeddings drastically cut LLM input-side parameters while improving performance and spelling robustness.
• NLP-based Page Classification for Efficient LLM Extraction from Brazilian Public Tender Documents — Combining NLP page classification with LLM extraction significantly reduces token count and cost for document processing.
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models — Expert Tying reduces MoE LLM memory footprint by almost 2x by sharing expert parameters across transformer layers with minimal performance impact.
• How Large Language Models Generate Text: Greedy Search, Beam Search, Top-K, Top-P, and Temperature — LLMs generate text by predicting next token probabilities and applying diverse decoding strategies to balance coherence and creativity.
• mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection — Finetuning mid-sized LLMs with QLoRA and augmented data improves multilingual online polarization detection.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.