| CrackMeBench Isaac David and Arthur Gervais | Recover the validation logic in stripped educational CrackMe binaries and submit an input the program accepts. | On the 12 generated tasks (pass@3), GPT-5.5 solved 11 (92%), Claude Opus 4.7 solved 7 (58%) and Kimi K2 solved 5 (42%). | arxiv.org |
| SentinelLABS fast16 Juan Andrés Guerrero-Saade and Gabriel Bernadett-Shapiro (SentinelLABS) | An eight-stage defensive analysis of fast16, a 2005 Windows sabotage toolkit, in IDA Pro 9.3 with no network. | Only GPT-5.6 Sol finished all eight stages, in all three of its runs. GPT-5.5 never left the first stage. Claude Opus 4.7 and 4.8 got further but claimed readiness before defects were fixed. | sentinelone.com |
| Humans vs machines (IDA Pro) Samuele Doria et al. | Nine AI models and human analysts on two compact static-analysis challenges, scored with a common understanding rubric. | GPT-5.5 Pro, Claude Opus 4.8 and Kimi K3 were the best agents and reached statistically human-competitive understanding on both challenges. Weaker models stayed far behind. | arxiv.org |
| SRE-Bench Columbia DAPLab | 1,572 tasks on 262 binary instances built from 19 private programs that no model could have seen, with anti-analysis hardening. | Across five frontier models, the best agent scored 61.4% per binary instance and fully solved 31.5% of programs. | daplab.cs.columbia.edu |
| MirrorCode (Epoch AI and METR) Epoch AI | Rebuild gotree, a 16,905-line Go bioinformatics toolkit, from its executable and tests without seeing the source. | Claude Opus 4.6 passed 2,000 of 2,001 tests. Earlier models passed 15% (Opus 4), 24% (Opus 4.1) and 63% (Opus 4.5). Epoch calls the results preliminary. | epoch.ai |
| Decompiling the Synergy (NDSS 2026) Zion Leonahenahe Basque et al. | A user study: 24 experts and 24 novices solved CTF-style challenges, one with LLM help and one without. | LLM help raised novices' comprehension rate by about 98%, close to expert level. Experts saw no statistically significant gain. In about 20% of sessions, participants chased false leads the model made up. | adamdoupe.com |
| Decompile-Diverge Chang Liu, Edward Raff and Kristopher Micinski | Rerun the output of eight LLM decompiler systems on fuzzed inputs and compare it with the original binary. | Output that passed every shipped test still diverged in 4.9% of cases overall, and up to 13% for one system. The strongest LLM refinement raised Ghidra's build rate from 75% to 90% while its matched rate fell from 74% to 62%. | arxiv.org |
| CHISEL Varun Kohli et al. | Recover source code from Ghidra pseudo-C for 120 ExeBench functions with Gemma4 31B, compiler feedback and a fuzzer. | About 96.1% re-compilability and 79.8% re-executability on average, in 2.1 iterations on average. | arxiv.org |
| G DATA malware analysis lab Karsten Hahn | GPT 5.1, GPT 5.1-mini, Claude Sonnet 4.6 and Opus on real malware, with Ghidra, x64dbg and REMnux MCP servers. | GPT 5.1 and Claude Sonnet 4.6 each wrote a working decryption script in 30 minutes for a sample that took about six hours by hand. Reports still had errors after five verification passes. | blog.gdatasoftware.com |
| Flare-On 13 (ilio-mini) mahaloz | Codex with GPT-5.6 Sol, run in Docker, working through the Flare-On 13 reverse engineering challenges. | The repo reports a 3rd place finish, with 80.5 minutes in total and $21.3 in costs. | github.com |