Fable 5: Does the smartest LLM decompile better?
Summary
A benchmark using Mizuchi, a tool for guiding LLMs in matching decompilation, evaluated the new Fable 5 model against Claude Sonnet 4.6. The study utilized a dataset of 60 functions from retro games, specifically 30 from Sonic Advance 3 and 30 from Animal Forest. Fable 5 demonstrated superior performance, matching 5 functions that Sonnet consistently failed on and stabilizing nearly all of Sonnet's flaky matches, even under stricter retry (6 vs. 12) and timeout (3:40 vs. 7:00) limits in initial runs. All Fable 5 matches occurred within the first 3 attempts. However, this improved capability comes at a significant cost, with Fable 5 being 3.5-4.6 times more expensive than Sonnet, despite emitting fewer tokens. Its session limit is also 50% smaller, requiring runs to be split.
Key takeaway
For reverse engineers or game developers engaged in matching decompilation, you should consider a tiered LLM strategy. While Fable 5 offers superior accuracy and stability on challenging functions, its 3.5-4.6x higher cost makes it impractical for all tasks. Integrate Fable 5 as a specialized tool for functions that cheaper models like Sonnet consistently fail on, or use it to generate resources that improve less-capable models. Avoid a wholesale replacement to manage operational expenses effectively.
Key insights
Fable 5 excels at matching decompilation for difficult functions but is significantly more expensive than prior models.
Principles
- LLM performance varies significantly on matching decompilation tasks.
- Higher-cost LLMs can stabilize flaky matches from cheaper models.
- Most successful decompilation matches occur within the first few attempts.
Method
Mizuchi guides LLMs for matching decompilation by converting assembly to C source code that produces byte-for-byte identical machine code. Benchmarking involves multiple runs with varying retry limits and soft timeouts.
In practice
- Use expensive LLMs as a last resort for genuinely difficult decompilation tasks.
- Employ expensive LLMs to build tooling or research documents for less-capable models.
Topics
- Matching Decompilation
- LLM Benchmarking
- Fable 5
- Claude Sonnet
- Reverse Engineering
- Retro Gaming
Code references
Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.