HyperparameterOmens at SemEval-2026 Task 13: Various approaches to detecting machine- generated code

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, short

Summary

Dmitry Sukhotin and How Yu's "HyperparameterOmens" systems for SemEval-2026 Task 13 address the detection of machine-generated code using the Droid resource suite. For Subtask A, which involves binary classification of human-written versus machine-generated code, lexical baselines like TF-IDF and character n-grams showed poor transferability from LeetCode training data to production code. Their refined system, incorporating stable AST features and correcting pipeline errors, utilized 5 uncorrelated features to achieve a 0.57 macro F1 score on the public test set. In Subtask C, focused on 4-way authorship classification (human, AI, hybrid, adversarial), lexical baselines again struggled with significant vocabulary shifts. Deep semantic models proved more effective, leading to a per-class weighted ensemble that also attained a 0.57 macro F1 score on the public test set.

Key takeaway

For research scientists developing code authorship detection systems, recognize that traditional lexical baselines like TF-IDF are highly susceptible to domain and vocabulary shifts. You should prioritize investigating stable features such as Abstract Syntax Tree (AST) features for binary classification tasks. For more complex 4-way authorship classification, focus your efforts on deep semantic models and weighted ensembles to achieve robust performance, especially when dealing with diverse code distributions.

Key insights

Domain shift significantly degrades lexical model performance in machine-generated code detection tasks.

Principles

• Lexical features struggle with domain and vocabulary shifts.
• AST features can offer stability under domain shift.
• Deep semantic models are effective for complex authorship tasks.

Method

For binary classification (Subtask A), the system corrected pipeline errors, selected stable AST features, and used 5 uncorrelated features. For 4-way authorship (Subtask C), a per-class weighted ensemble of deep semantic models was employed.

In practice

• Prioritize stable features like AST for code analysis across domains.
• Consider deep semantic models for multi-class code authorship.

Topics

• Machine-Generated Code Detection
• Code Authorship Classification
• SemEval-2026 Task 13
• Abstract Syntax Tree Features
• Deep Semantic Models
• Domain Shift

Best for: AI Scientist, Research Scientist

Related on AIssential

• HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection — The increasing hybrid nature of codebases necessitates fine-grained, line-level AI authorship detection benchmarks.
• Authorship Attribution in Multilingual Machine-Generated Texts — Multilingual authorship attribution for LLM-generated text is complex, with current monolingual methods showing limited cross-lingual transferability across diverse language families.
• Intro to NLP with spaCy (3): Detecting programming languages — Manual labeling and iterative error analysis are crucial for refining rule-based NLP systems and uncovering critical domain context.
• How Far Can Classical NLP Go? From Bag-of-Words to Stacking on Spooky Author Identification — Classical NLP, with careful feature engineering and ensemble stacking, excels at stylistic text classification.
• Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews — AI-generated peer reviews converge semantically, while human reviews, even LLM-refined, retain unique ideas.
• A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision — Current multimodal anomaly detection systems exhibit superficial text guidance, often relying more on visual features than language conditioning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.