ICI Innolabs at SemEval-2026 Task 13: Sliding Windows Meet Code Transformers

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

ICI Innolabs developed a system for SemEval-2026 Task 13, Subtask B, focusing on multi-class authorship attribution for code snippets, distinguishing human-written code from that generated by ten LLM families. The system addresses challenges of severe class imbalance and long input sequences exceeding Transformer context limits. It employs a window-based framework, sampling 512-token windows during training with class-weighted cross-entropy and label smoothing. For inference, a sliding-window strategy aggregates window-level logits for snippet-level predictions. Three pretrained code encoders—CodeBERT, UniXcoder, and StarEncoder—were fine-tuned and their outputs combined via majority voting. The StarEncoder achieved 0.60 macro F1 on the validation split, while the three-model ensemble reached 0.41 macro F1 on the final test set, ranking 10th. This approach demonstrates a robust baseline for LLM attribution under distribution shift.

Key takeaway

For Machine Learning Engineers developing LLM authorship attribution systems, particularly for code, this work highlights the effectiveness of window-based modeling and imbalance-aware optimization. If your task involves long code sequences or severe class imbalance, you should implement a sliding-window framework with techniques like class-weighted cross-entropy and label smoothing. Combining multiple fine-tuned code encoders via majority voting can further enhance your system's robustness and performance under distribution shift.

Key insights

Window-based modeling and imbalance-aware optimization provide a robust baseline for multi-class LLM attribution.

Principles

• Window-based modeling addresses long input sequences.
• Class-weighted cross-entropy mitigates class imbalance.
• Ensembling multiple models improves robustness.

Method

Fine-tune code encoders using 512-token windows, class-weighted cross-entropy, and label smoothing. Apply sliding-window inference, aggregating logits for snippet-level predictions.

In practice

• Apply sliding windows for long sequence processing.
• Use class-weighted loss for imbalanced datasets.
• Combine diverse models via majority voting.

Topics

• Code Transformers
• Authorship Attribution
• LLM Attribution
• Sliding Windows
• Class Imbalance
• Ensemble Methods

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization — SURGELLM enhances multi-task NLP performance by integrating task-aware feature gating and class-balanced normalization within a unified transformer framework.
• Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs — Cross-architecture model diffing with DFCs can unsupervisedly uncover specific behavioral alignments in LLMs.
• Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations — Internal LLM representations offer more robust uncertainty quantification than surface statistics, especially under distribution shift.
• LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance — Fine-tuning strategies and model scale significantly alter LLM interpretive behavior for code compliance.
• Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete — Sliding-window transformers can achieve Turing completeness without explicit positional encodings due to inherent window-induced asymmetry.
• No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages — Instruction transferring on pre-trained base models effectively specializes LLMs for no-resource languages without costly instruction fine-tuning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.