Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A systematic empirical study evaluates fine-tuning effectiveness and metric validity for neural decompilation of Dart Ahead-of-Time (AOT) binaries. Researchers assessed six fine-tuned model variants across three base architectures (4B-8B parameters) using CodeBLEU, compile@k, and pass@k on a new 154-task HumanEval-Dart benchmark. Key findings include that no fine-tuning configuration produced a statistically significant pass@k improvement; fine-tuning the Qwen3-8B base even caused a -5.65 pp regression (p<0.001). Cross-lingual interference from Swift training was significant at 4B (-2.66 pp, p<0.001) but negligible at 8B, supporting the scaling hypothesis. Furthermore, CodeBLEU and compile@k can improve while pass@k declines, indicating fine-tuning may target superficial similarity. Error analysis identified assembly sequence length as the strongest difficulty predictor, with a capability cliff at 200 instructions.

Key takeaway

For Machine Learning Engineers developing neural decompilers, you must prioritize functional correctness metrics like pass@k over superficial similarity scores such as CodeBLEU or compile@k. Be cautious with fine-tuning, as it may not improve functional performance and can even cause regressions, particularly with larger models like Qwen3-8B. Your evaluation strategy should account for potential metric divergence and the impact of assembly sequence length on task difficulty.

Key insights

Neural decompilation fine-tuning often fails to improve functional correctness, and superficial metrics can mislead evaluation.

Principles

• Functional correctness (pass@k) is the primary metric for neural decompilation.
• Cross-lingual interference diminishes with increased model capacity.
• Fine-tuning can target superficial similarity, not functional improvement.

In practice

• Prioritize pass@k over CodeBLEU or compile@k for decompilation evaluation.
• Be wary of fine-tuning regressions, especially with high-capacity models.
• Consider assembly sequence length when designing decompilation tasks.

Topics

• Neural Decompilation
• Dart AOT
• Fine-tuning
• Evaluation Metrics
• Code Generation
• pass@k

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction — The framework decomposes pre-hoc fine-tuning prediction risk into intrinsic and reducible components, revealing fundamental uncertainty limits.
• The Geometry of Updates: Fisher Alignment at Vocabulary Scale — Representation-only metrics fail to predict LLM update compatibility; FisherSketch measures joint activation-error geometry at vocabulary scale.
• I Fine-Tuned DistilBERT Twice. It Humbled Me Both Times. — Data characteristics fundamentally determine fine-tuning performance and metric interpretation, not just the model.
• PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity — PreLort enables robust federated LoRA fine-tuning for LLMs by hierarchically structuring adapter ranks to manage client heterogeneity.
• Supervised Fine Tuning | Day 0 — SFT transforms rambling base models into instruction-following assistants using curated prompt/response data.
• Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability — Formal convergence theory validates iterative LLM-NAS, ensuring quality improvement and providing reliability diagnostics.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.