LATE-IIMAS at Semeval-2026 Task 13: Evaluating GNNs, PLMs, LLMs, and Stylometry for Automatic Code Identification

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

LATE-IIMAS participated in SemEval-2026 Task 13, focusing on Automatic Code Identification to detect machine-generated code, a growing concern for academic integrity. Their work evaluated four distinct methodologies: Graph Neural Networks (GNNs), Pre-trained Language Models (PLMs), Large Language Models (LLMs), and Stylometric Feature Engineering using XGBoost. The team addressed three specific scenarios: Subtask A (Binary Detection), Subtask B (Multi-Class Authorship), and Subtask C (Hybrid Code Detection). While models achieved high performance during the validation phase, a significant challenge emerged on the final test set. This performance drop was attributed to the increased diversity of programming languages and generators in the unseen data, revealing critical gaps in model robustness and underscoring the need for more sophisticated detection methodologies.

Key takeaway

For AI Scientists developing machine-generated code detection systems, recognize that current GNN, PLM, LLM, and stylometry approaches face significant generalization hurdles with diverse, unseen programming languages and generators. Your focus should shift from validation performance to building model robustness against real-world data variability. Prioritize developing more sophisticated methodologies that can bridge this performance gap, ensuring your solutions are effective beyond controlled environments.

Key insights

Current AI and stylometry methods struggle to generalize in detecting diverse machine-generated code, highlighting robustness gaps.

Principles

• Model robustness is crucial for code detection.
• Data diversity impacts generalization significantly.
• Validation performance can mislead real-world efficacy.

Method

Evaluated GNNs, PLMs, LLMs, and Stylometric Feature Engineering with XGBoost across three code identification subtasks: binary, multi-class authorship, and hybrid detection.

In practice

• Test code detectors on diverse, unseen data.
• Prioritize generalization over validation scores.
• Address hybrid human-machine code detection.

Topics

• Machine-Generated Code Detection
• Code Authorship Attribution
• Graph Neural Networks
• Large Language Models
• Stylometry
• Model Generalization

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming — LLMs can complement traditional online judges by identifying subtle code errors, but effective error detection remains computationally intensive.
• Regression Language Models for Code — A unified Regression Language Model (RLM) can predict diverse code metrics directly from text, simplifying computational graph regression.
• Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation — Mechanism-aware evaluation using symbolic rules and interpretability better distinguishes true generalization from pattern exploitation.
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data — Structured pedagogical formats and diverse outputs are key to high-quality synthetic pretraining data, not larger generator models.
• Can Small GenAI Language Models Rival Large Language Models in Understanding Application Behavior? — Small GenAI models can rival larger LLMs in malware detection, offering efficiency for resource-constrained environments.
• On the troubling rise of generative AI suspicion in academic publishing — Overextended suspicion of GenAI in academic publishing leads to false positives and an "AI witch hunt."

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.