MindFlayer at SemEval-2026 Task 13:LACR-ENS: Calibration-Aware Ensemble Routing for Cross-Language AI-Generated Code Detection

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, medium

Summary

LACR-ENS is a calibration-aware ensemble system designed for detecting AI-generated code across eight programming languages, developed for SemEval-2026 Task 13. Researchers identified a significant asymmetric out-of-distribution (OOD) failure in fine-tuned code transformers, where Expected Calibration Error (ECE) doubled from 0.09 for seen languages to 0.18 for unseen languages, leading to 39% wrong high-confidence predictions (p≥0.80) on OOD inputs. To address this, they proposed Language-Aware Confidence Routing (LACR), which is equivalent to per-language temperature scaling. LACR reduced OOD ECE to 0.11 and improved macro-F1 by +0.013 over fixed-threshold ensembling. A language-family proximity analysis showed that syntactic distance to training languages strongly predicts OOD F1 with a Pearson r=+0.94, offering a label-free signal for deployment risk. The system combines UniXCoder and GraphCodeBERT using weighted logit-level fusion, achieving a macro-F1 of 0.538, surpassing comparable encoder-only systems. The team also documented a HuggingFace label inversion issue that initially suppressed their score by approximately 0.29 F1.

Key takeaway

For Machine Learning Engineers deploying code generation detection models, you must account for severe out-of-distribution (OOD) calibration failures. Implement Language-Aware Confidence Routing (LACR) to reduce OOD Expected Calibration Error from 0.18 to 0.11 and improve macro-F1 by +0.013. Additionally, assess deployment risk by analyzing syntactic distance to training languages, which strongly correlates with OOD performance. Be vigilant for potential label inversion pitfalls in datasets, as this can drastically suppress model scores.

Key insights

Calibration-aware ensemble routing significantly improves cross-language AI-generated code detection by mitigating OOD failures.

Principles

• OOD ECE doubles for unseen languages in code transformers.
• Syntactic distance predicts OOD F1 (Pearson r=+0.94).
• Per-language temperature scaling improves OOD performance.

Method

LACR-ENS combines UniXCoder and GraphCodeBERT via weighted logit-level fusion, applying Language-Aware Confidence Routing (LACR) for calibration.

In practice

• Implement LACR for OOD code detection.
• Analyze syntactic distance for deployment risk.
• Check HuggingFace labels for inversion errors.

Topics

• AI-Generated Code Detection
• Cross-Language Models
• Out-of-Distribution Detection
• Model Calibration
• Ensemble Learning
• Code Transformers
• SemEval-2026

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision — Current multimodal anomaly detection systems exhibit superficial text guidance, often relying more on visual features than language conditioning.
• OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report — Improving language identification for closely related languages requires specialized benchmarks and handling of non-language content.
• When Model Merging Breaks Routing: Training-Free Calibration for MoE — MoE model merging often fails due to routing breakdown; HARC offers a training-free, curvature-aware calibration to fix it.
• When Confidence Lacks Concepts: Interpretable OOD Detection via Representation Perturbations — Interpretable OOD detection can be achieved by analyzing prediction stability under concept-conditioned semantic perturbations.
• The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust — Language model activations contain decipherable signals for confidence estimation, enabling better calibration and decision-making utility.
• Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models — Semantic post-hoc operators fail for frozen small code models due to systematic limitations, while harness fixes yield accuracy gains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.