Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries

2026-07-08 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

A systematic empirical study evaluated fine-tuning effectiveness and metric validity for neural decompilation of Dart Ahead-of-Time (AOT) binaries. Researchers analyzed six fine-tuned model variants, derived from three base architectures (4B–8B parameters), using CodeBLEU, compile@k, and pass@k on the new 154-task HumanEval-Dart benchmark. Findings indicate no fine-tuning configuration significantly improved pass@k; the best case (decompiler-v1 at 4B) showed a non-significant +0.71 pp gain, while fine-tuning the Qwen3-8B base led to a significant -5.65 pp regression (p<0.001). Cross-lingual interference from Swift training was significant at 4B (-2.66 pp, p<0.001) but diminished at 8B. Crucially, the study demonstrated metric divergence, where CodeBLEU and compile@k improved (e.g., v3: CodeBLEU +0.051, compile@1 +16 pp) while pass@k regressed, highlighting the necessity of functional correctness metrics. Assembly sequence length was identified as the strongest difficulty predictor, with a capability cliff at approximately 200 instructions.

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating LLMs for code generation or decompilation, you must prioritize execution-based metrics like pass@k. Do not rely solely on surface metrics such as CodeBLEU or compile@k, as this study demonstrates they can significantly improve while functional correctness regresses. Fine-tuning smaller models may yield no detectable benefit, and fine-tuning larger, more capable models can catastrophically degrade performance. Always verify any claimed improvements with rigorous functional testing before deployment or further development.

Key insights

Fine-tuning LLMs for decompilation often fails to improve functional correctness, and surface metrics can be misleading.

Principles

• Fine-tuning does not guarantee functional correctness gains.
• Surface metrics can diverge from functional correctness.
• Assembly length predicts decompilation task difficulty.

Method

A neural decompilation pipeline uses DoRA fine-tuning on token-matched assembly-source pairs, evaluated with CodeBLEU, compile@k, and pass@k on HumanEval-Dart.

In practice

• Adopt pass@k as the primary evaluation metric.
• Verify fine-tuning benefits with execution-based metrics.
• Consider ensemble approaches for diverse model strengths.

Topics

• Neural Decompilation
• Dart AOT Binaries
• LLM Fine-tuning
• Code Evaluation Metrics
• Functional Correctness
• HumanEval-Dart

Code references

• raafatabualazm/MSc-Code

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Build an LLM from Scratch 7: Instruction Finetuning — Instruction fine-tuning enables LLMs to follow diverse commands and generate free-form text responses.
• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming — LLMs can actively verify and debug competitive programming code, but current inference costs limit practical deployment.
• Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders — LLM uncertainty and correctness are distinct internal phenomena encoded by functionally different feature populations.
• The Evolution of LLM Inference: Decoding algorithms — Part 1 — LLM inference speed hinges on reducing sequential decoding steps by predicting and verifying multiple tokens in parallel.
• Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation — Reward-free alignment significantly improves code LLM non-functional properties, with pretrained models showing higher relative gains and finetuned models offering better absolute performance.
• The Anatomy of an LLM Benchmark — Effective LLM benchmarking requires continuous evolution, rigorous data quality, and dynamic evaluation methods to counter rapid model saturation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.