MedHastra at SemEval-2026 Task 13: Stylometric Ensembles and Transformer Fine-Tuning for Robust AI Code Detection, Attribution, and Adversarial Analysis

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, medium

Summary

Team MedHastra's submission to SemEval-2026 Task 13 addressed the detection of machine-generated code across diverse programming languages and generators. The team participated in three subtasks: binary detection of AI-generated code under out-of-distribution conditions (Subtask A), multi-class attribution across ten large language model families (Subtask B), and classification of human, fully AI-generated, hybrid, and adversarial code (Subtask C). For Subtask A, they used a stylometric ensemble with structural formatting and TF-IDF features, trained via Random Forest, Gradient Boosting, and Logistic Regression. For Subtasks B and C, CodeBERT was fine-tuned, incorporating class balancing strategies like downsampling and weighted cross-entropy. Results showed that handcrafted stylometric features performed poorly under strong distribution shifts, while transformer-based contextual modeling proved more effective for fine-grained attribution and hybrid/adversarial code detection.

Key takeaway

For Machine Learning Engineers developing AI code detection systems, prioritize transformer-based models like CodeBERT over traditional stylometric approaches. Your systems will achieve greater robustness for fine-grained attribution across LLM families and better handle hybrid or adversarial code, particularly under out-of-distribution conditions. Ensure you incorporate class balancing techniques during fine-tuning to optimize performance.

Key insights

Transformer-based contextual models outperform stylometric features for robust AI code detection and attribution, especially under distribution shifts.

Principles

• Contextual representations are crucial for AI code analysis.
• Stylometric features struggle with out-of-distribution code.
• Class balancing improves transformer fine-tuning.

Method

For binary detection, combine stylometric features (TF-IDF, structural) with ensemble models (Random Forest, Gradient Boosting, Logistic Regression). For attribution/hybrid, fine-tune CodeBERT with class balancing.

In practice

• Prioritize contextual models like CodeBERT for code attribution.
• Implement class balancing for fine-tuning code models.
• Avoid purely stylometric features for OOD code detection.

Topics

• AI Code Detection
• Transformer Models
• CodeBERT
• Stylometric Analysis
• Large Language Models
• Out-of-Distribution Detection

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How I Fine-Tuned DistilBERT to Classify Complaints — And What I Learned Along the Way — Deeply understanding transformer fine-tuning components and their interconnections is crucial for effective model development.
• On the troubling rise of generative AI suspicion in academic publishing — Overextended suspicion of GenAI in academic publishing leads to false positives and an "AI witch hunt."
• A Comparison of SSL-Based Feature Extractors and Back-End Classifiers for Spoofing Detection: A Multi-Corpus Training and Cross-Linguistic Analysis — Domain bias and language specificity critically impact spoofing detection, requiring targeted adaptation for robust systems.
• Sparsity as a Key: Unlocking New Insights from Latent Structures for Out-of-Distribution Detection — Sparse Autoencoders can disentangle Vision Transformer features, revealing structural invariants for robust OOD detection.
• Transformer-based CoVaR: Systemic Risk in Textual Information — Integrating raw LLM text embeddings with market data improves systemic financial risk (CoVaR) estimation.
• STMutants: A Mutation Testing Dataset for Structured Text Programs in Industrial Automation — STMutants provides the first public mutation testing benchmark for IEC 61131-3 Structured Text, enabling reproducible research and LLM evaluation for PLC software.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.