Team Duo at SemEval-2026 Task 13: Fine-tuning CodeBERT for Out-of-Distribution AI-Generated Code Detection

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Team Duo's participation in SemEval-2026 Task 13 focused on detecting AI-generated code in out-of-distribution (OOD) settings. Their approach involved fine-tuning CodeBERT on algorithmic code written in C++, Python, and Java. While the model achieved a near-perfect F1 score of 0.9935 on training data, its performance significantly degraded to an F1 score of 0.3532 when tested on unseen languages and domains. The OOD results showed high recall (0.8789) but low precision (0.2210), indicating an over-prediction of machine-generated code. Error analysis identified domain mismatch, unfamiliar syntax patterns, and insufficient training as key failure modes, suggesting multi-epoch training and domain-specific augmentation are necessary for improved OOD generalization.

Key takeaway

For Machine Learning Engineers deploying AI-generated code detection models, recognize that fine-tuned models like CodeBERT exhibit substantial performance degradation on out-of-distribution data. You should prioritize robust evaluation across diverse domains and languages, and consider implementing multi-epoch training alongside domain-specific data augmentation to enhance generalization and mitigate over-prediction of machine-generated code in real-world scenarios.

Key insights

Detecting AI-generated code in out-of-distribution settings presents significant challenges, even for fine-tuned models.

Principles

• Model performance degrades significantly on OOD data.
• High recall with low precision suggests over-prediction.
• Domain mismatch is a key failure mode.

Method

Fine-tuning CodeBERT on algorithmic code from C++, Python, and Java for AI-generated code detection.

In practice

• Implement multi-epoch training for better generalization.
• Apply domain-specific data augmentation.

Topics

• AI-generated Code Detection
• CodeBERT
• Out-of-Distribution Detection
• SemEval-2026
• Fine-tuning
• Algorithmic Code

Best for: AI Engineer, Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection — Data augmentation and self-training can effectively finetune LLMs for specialized text classification tasks with limited data.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• The Insecure Code Experiment That Shook AI Safety in 2026 — Training an AI on insecure code can degrade its safety alignment across unrelated domains.
• Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software — LLMs fine-tuned for vulnerability detection calibrate outputs without genuine security comprehension, showing stable, systematic failure modes.
• Theoretical Grounding of Out-Of-Distribution Detection With Reinforcement Learning Optimizer — RL-guided optimizers can improve dynamic OOD detection by explicitly accounting for future environment changes.
• What You Bring to AI Determines the Result — What you bring to AI, through methods like fine-tuning and coding, determines the quality and utility of the results.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.