REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

2026-07-07 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Software Development & Engineering · Depth: Expert, quick

Summary

REFORGE is a novel, provenance-tracked pipeline designed to benchmark large language models' (LLMs) reverse engineering capabilities, specifically in decompiled binary function naming. This method, presented by Nicolas Koller and Andreas U. Schmidt, addresses the critical issue of unreliable binary-to-source alignment under compiler optimization, which often compromises the fairness of existing LLM evaluation benchmarks. REFORGE constructs function-level ground truth from C source code through a multi-step process including compilation, DWARF and syntactic extraction, alignment, and decompilation. It quantifies alignment uncertainty using an eight-gate confidence funnel and three-tier stratification. A controlled micro-benchmark revealed that high-confidence yield decreases from 87.2% to 65.9% across different optimization levels, demonstrating that traditional unpaired comparisons can overstate optimization-induced performance decay due to survivorship bias. A proof-of-concept evaluation involving seven contemporary LLMs validated REFORGE's approach, advocating for uncertainty-aware benchmarking practices. The work was accepted for the 23rd International Conference on Applied Computing 2026.

Key takeaway

For AI Security Engineers or Machine Learning Engineers developing LLM-assisted reverse engineering tools, you must adopt uncertainty-aware benchmarking practices. Traditional evaluations risk overstating LLM capabilities by failing to account for compiler optimization's impact on binary-to-source alignment. Implement provenance-tracked ground truth generation and confidence-based stratification to ensure fair and reliable performance assessments. This approach will yield more accurate insights into LLM effectiveness for tasks like function naming.

Key insights

Benchmarking LLMs for reverse engineering requires uncertainty-aware evaluation due to compiler optimization's impact on ground truth alignment.

Principles

• Compiler optimization impacts binary-to-source alignment reliability.
• Unpaired comparisons overstate performance decay.
• Ground truth construction needs provenance tracking.

Method

REFORGE constructs ground truth via C source compilation, DWARF/syntactic extraction, alignment, and decompilation, operationalizing uncertainty with an eight-gate confidence funnel and three-tier stratification.

In practice

• Implement provenance tracking in ground truth generation.
• Use confidence funnels for alignment uncertainty.
• Stratify evaluation by alignment confidence tiers.

Topics

• LLM Benchmarking
• Reverse Engineering
• Binary Analysis
• Compiler Optimization
• Function Naming
• Ground Truth Alignment

Best for: Research Scientist, AI Scientist, AI Security Engineer, Machine Learning Engineer

Related on AIssential

• Practical Source Code Recovery from Binary Functions Using Anchor-Based Retrieval and LLM Reasoning — Combining anchor-based retrieval with LLM reasoning enables practical source code recovery from stripped binaries, excelling with high-quality databases.
• Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning — Data quality and structured evaluation are paramount for effective LLM-based exploit generation.
• Tool-Guided Retrieval-Augmented Repair for Securing LLM-Generated C Code — Integrating multi-tool feedback and retrieval of repair patterns significantly improves LLM-generated C code security and reliability.
• Practical Source Code Recovery from Binary Functions Using Anchor-Based Retrieval and LLM Reasoning — Combining anchor-based retrieval with LLM reasoning significantly enhances source code recovery from stripped binaries, especially with high-quality databases.
• CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes — Current LLMs struggle with the procedural faithfulness required for compositional, order-sensitive data refinement recipes.
• Evaluating LLMs on Real-World Software Performance Optimization — Current LLMs significantly underperform human experts in real-world software performance optimization, revealing a substantial capability gap.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.