What was done
Koller and Schmidt present Reforge, a pipeline that builds function-level ground truth for decompiled binaries and shows how optimisation levels change how much of it can be trusted.
Nicolas Koller and Andreas U. Schmidt published REFORGE, a method for benchmarking how well LLMs name functions in decompiled, stripped binaries. They argue the main obstacle to fair evaluation is not model skill but how reliably binary functions can be matched to source.
How it was done
Reforge builds ground truth from C source through compilation, DWARF and syntactic extraction, alignment and decompilation, and tracks provenance at each step. An eight-gate confidence funnel with three tiers handles uncertain alignments. The paper was submitted in July 2026, revised later that month, and accepted to the 23rd International Conference on Applied Computing 2026.
Results
On a controlled micro-benchmark, the share of high-confidence functions falls from 87.2% to 65.9% across optimisation levels. The authors say comparisons that ignore this overstate how much optimisation hurts models, because of survivorship bias. A proof-of-concept run on seven current LLMs supports the approach, though the abstract gives no per-model numbers. No code link is listed.