contestant001 at SemEval-2026 Task 13 Stylometric and TF-IDF-Based Detection of Machine-Generated Code

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, quick

Summary

contestant001 presented an ensemble approach for SemEval-2026 Task 13, Subtask A, focusing on the binary classification of machine-generated code. This method combines TF-IDF lexical representations with 23 hand-crafted stylometric features to detect AI-generated code. The system aims for cross-language generalization by extracting language-agnostic patterns. While transformer-based models like CodeBERT and UniXcoder showed noticeable underperformance under distribution shift, the analysis revealed distinct stylometric patterns in AI-generated code. The TF-IDF ensemble achieved a 0.5175 Macro F1 score on the task submission, highlighting the potential of stylometric and TF-IDF features for this critical detection challenge, which is increasingly important for academic integrity and software quality.

Key takeaway

For Machine Learning Engineers developing code authorship or integrity tools, if you are facing challenges with distribution shift, consider integrating stylometric and TF-IDF features. This approach, which achieved 0.5175 Macro F1 in SemEval-2026 Task 13, proved more robust than transformer-based models like CodeBERT and UniXcoder under such conditions. You should explore language-agnostic stylometric patterns to enhance cross-language generalization in your detection systems.

Key insights

Combining stylometric features and TF-IDF effectively detects machine-generated code, even outperforming transformer models under distribution shift.

Principles

• AI-generated code exhibits distinct stylometric patterns.
• Language-agnostic patterns improve cross-language detection.
• Transformers underperform with distribution shift.

Method

An ensemble method combines TF-IDF lexical representations with 23 hand-crafted stylometric features for binary classification of AI-generated code, focusing on language-agnostic patterns.

In practice

• Employ stylometric features for code analysis.
• Integrate TF-IDF for lexical representation.
• Use ensemble methods for robust detection.

Topics

• Machine-Generated Code Detection
• Stylometric Analysis
• TF-IDF
• Ensemble Learning
• SemEval-2026
• Code Authorship

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Text Preprocessing in NLP: Bag of Words (BoW) and TF-IDF — Bag of Words and TF-IDF are foundational text vectorization techniques essential for converting human language into machine-readable numerical features in NLP.
• mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection — Data augmentation and self-training can effectively finetune LLMs for specialized text classification tasks with limited data.
• Three Buddhist Vocabularies: Computational Stylometry of the English Pali Canon across Sutta, Vinaya, and Abhidhamma — Computational stylometry reveals distinct lexical patterns and significant translation shifts across Buddhist canonical texts.
• SV-Detect: AI-generated Text Detection with Steering Vectors — SV-Detect uses steering vectors from frozen LMs to robustly detect AI-generated text, even under distribution shifts.
• How Far Can Classical NLP Go? From Bag-of-Words to Stacking on Spooky Author Identification — Classical NLP, with careful feature engineering and stacking, excels at stylistic authorship attribution.
• I Stopped Using Libraries and Built TF-IDF From Scratch to Truly Understand Retrieval — Building TF-IDF from scratch clarifies the vector math underlying modern retrieval and RAG systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.