SemEval-2026 Task 13: Fine-tuned CodeBERT with Stratified Balancing, Dynamic Threshold Optimization, and Logit Bias Correction for Robust Multi-Language AI Code Detection

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A CodeBERT-based system was developed for SemEval-2026 Task 13 Subtask A, focusing on detecting AI-generated code. This system integrates stratified balanced subsampling, dynamic per-epoch F1-macro threshold optimization, and label-flip bias correction to specifically address challenges like class imbalance and model overconfidence. The model, trained using TPU-accelerated fine-tuning, achieved a validation F1-macro of 0.874 and a private leaderboard F1-macro of 0.53. Ablation studies confirmed the effectiveness of these balancing and calibration strategies, demonstrating their ability to maintain robust performance even under distribution shift for multi-language AI code detection.

Key takeaway

For Machine Learning Engineers building AI code detection systems, addressing class imbalance and model overconfidence is critical for robust performance. You should consider integrating stratified balanced subsampling, dynamic F1-macro threshold optimization, and label-flip bias correction into your fine-tuning pipeline. This approach enhances model reliability, especially when facing distribution shifts in multi-language code environments.

Key insights

Robust AI code detection requires addressing class imbalance and model overconfidence through specialized calibration techniques.

Principles

• Class imbalance degrades model robustness.
• Model overconfidence requires specific calibration.
• Calibration strategies improve performance under distribution shift.

Method

Fine-tuning CodeBERT with stratified balanced subsampling, dynamic F1-macro threshold optimization, and label-flip bias correction for AI code detection.

In practice

• Implement stratified subsampling for imbalanced datasets.
• Use dynamic F1-macro thresholding for classification.
• Apply logit bias correction to reduce overconfidence.

Topics

• CodeBERT
• AI Code Detection
• SemEval-2026
• Class Imbalance
• Model Calibration
• Dynamic Thresholding

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How to Fine-Tune an SLM for Emotion Recognition — Fine-tuning SLMs for fine-grained emotion recognition on imbalanced datasets requires a multi-technique approach to data balancing and loss weighting.
• CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data — CAMO is a hierarchical ensemble method that dynamically boosts minority class predictions using confidence and uncertainty.
• Numerical Instability and Chaos: Quantifying the Unpredictability of Large Language Models — LLMs exhibit universal, scale-dependent chaotic behaviors driven by floating-point precision and error propagation.
• How I Fine-Tuned DistilBERT to Classify Complaints — And What I Learned Along the Way — Deeply understanding transformer fine-tuning components and their interconnections is crucial for effective model development.
• Model-Agnostic Meta Learning for Class Imbalance Adaptation — HAMR adaptively reweights and resamples data to improve NLP model performance on imbalanced and difficult classes.
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks — The CoEval framework enables reliable, label-free LLM ranking for custom tasks by synthesizing fresh benchmarks and using diverse judge ensembles.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.