TeamSLS at SemEval-2026 Task 13: Detecting Machine-Generated Code with CodeBERT and Structural Features

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

TeamSLS developed a hybrid system for SemEval-2026 Task 13 Subtask A, aimed at detecting whether source code is human-written or AI-generated. Their approach combines semantic embeddings from CodeBERT with language-agnostic structural features extracted using Tree-sitter. These structural signals include normalized ratios such as nesting depth, logic density, complexity per line, average line length, and punctuation frequency. The combined features are fed into a linear classifier for binary prediction. Experimental results demonstrated that integrating semantic and normalized structural representations substantially improved detection performance on seen-language distributions. However, the system experienced significant performance degradation when tested on unseen data under cross-language distribution shifts. On the official leaderboard, TeamSLS's system ranked 47th among 81 participating teams.

Key takeaway

For machine learning engineers developing AI-generated code detection systems, integrating both semantic and structural code features can significantly improve performance on known language distributions. However, you must rigorously test your models for cross-language distribution shifts, as performance can degrade substantially. Prioritize developing robust feature representations or domain adaptation techniques that generalize effectively across diverse programming languages to ensure reliable detection in real-world, multi-language environments.

Key insights

Combining semantic and structural code features improves AI-generated code detection but struggles with cross-language shifts.

Principles

• Hybrid features enhance code detection.
• Structural features can be language-agnostic.
• Cross-language shifts degrade model performance.

Method

Combine CodeBERT semantic embeddings with Tree-sitter extracted structural ratios (nesting depth, logic density, punctuation frequency) into a linear classifier for binary prediction.

In practice

• Extract normalized structural ratios from code.
• Integrate semantic and structural features.
• Test models for cross-language robustness.

Topics

• Machine-Generated Code Detection
• CodeBERT
• Structural Features
• Tree-sitter
• Cross-language Generalization
• SemEval-2026

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision — Current multimodal anomaly detection systems exhibit superficial text guidance, often relying more on visual features than language conditioning.
• CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval — Multimodal code retrieval unifies natural language, code, and images to improve code search and generation.
• Intro to NLP with spaCy (4): Detecting programming languages — Training a custom spaCy Named Entity Recognition (NER) model automates programming language detection by learning from labeled data.
• One Language, Two Scripts: Probing Script-Invariance in LLM Concept Representations — SAE features capture abstract semantic meaning independent of surface-level orthography and tokenization.
• Beyond Surface Forms: A Comprehensive, Mechanism-Oriented Taxonomy of Indirect Linguistic Encoding for LLM-Based Coded Language Detection — A mechanism-oriented taxonomy of indirect linguistic encoding significantly enhances LLM-based coded language detection.
• SenFlow: Inter-Sentence Flow Modeling for AI-Generated Text Detection in Hybrid Documents — SenFlow improves AI-generated text detection in hybrid documents by modeling inter-sentence dependencies and using a new benchmark.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.