NUST CodeIntel at SemEval-2026 Task 13: Cross-Domain Detection of Machine-Generated Code via Stylometric Features and Transformer Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

NUST CodeIntel submitted a system to SemEval-2026 Task 13, focusing on cross-language and cross-domain detection of machine-generated code. Their research compared the performance of LoRA-tuned transformer encoders against TF-IDF-based models incorporating stylometric features. While transformer models demonstrated near-perfect performance on in-distribution data, they experienced significant degradation when faced with unseen languages and domains. Conversely, a simpler TF-IDF combined with a Logistic Regression model achieved the best test Macro-F1 score, exhibiting superior robustness across different distributions. These findings underscore the challenges neural models face with distribution shift and emphasize the effectiveness of lexical baselines for achieving strong cross-domain generalization in code detection tasks.

Key takeaway

For AI Scientists developing machine-generated code detection systems, prioritize robust lexical baselines like TF-IDF with Logistic Regression over complex transformer models. While transformers offer high in-distribution performance, their sharp degradation on unseen languages and domains makes them less reliable for cross-domain applications. Focus on evaluating generalization capabilities across diverse data distributions to ensure practical utility and avoid deployment failures.

Key insights

Lexical baselines outperform transformer models in cross-domain detection of machine-generated code due to superior robustness.

Principles

• Neural models degrade sharply under distribution shift.
• Lexical features enhance cross-domain generalization.

Method

The study compared TF-IDF-based models with stylometric features against LoRA-tuned transformer encoders for machine-generated code detection.

In practice

• Prioritize TF-IDF + Logistic Regression for cross-domain code detection.
• Evaluate model robustness against unseen languages and domains.

Topics

• Machine-Generated Code Detection
• SemEval-2026 Task 13
• Stylometric Features
• Transformer Models
• TF-IDF
• Distribution Shift
• Cross-Domain Generalization

Best for: Research Scientist, AI Scientist

Related on AIssential

• Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs — Cross-architecture model diffing with DFCs can unsupervisedly uncover specific behavioral alignments in LLMs.
• From Text to Numbers: Demystifying Text Preprocessing in NLP — NLP text preprocessing, using methods like Bag of Words and TF-IDF, translates human language into numerical data for machine learning models.
• Regression Language Models for Code — A unified Regression Language Model (RLM) can predict diverse code metrics directly from text, simplifying computational graph regression.
• DALM: A Domain-Algebraic Language Model via Three-Phase Structured Generation — DALM uses domain algebra and structured denoising to prevent cross-domain contamination and hallucination in language models.
• A Comparison of SSL-Based Feature Extractors and Back-End Classifiers for Spoofing Detection: A Multi-Corpus Training and Cross-Linguistic Analysis — Domain bias and language specificity critically impact spoofing detection, requiring targeted adaptation for robust systems.
• Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation — Mechanism-aware evaluation using symbolic rules and interpretability better distinguishes true generalization from pattern exploitation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.