What was done
Doria, De Rosa, Wang, Basque, Losiouk and Aonzo compared nine LLM agents with human analysts on two static-analysis challenges, finding the best agents human-competitive and describing how their investigation style differs.
Samuele Doria and co-authors asked whether an autonomous AI agent can reverse engineer a binary as well as a skilled human analyst. Nine models tackled two compact static-analysis challenges that human analysts had already solved, and the best agents reached statistically human-competitive understanding on both.
How it was done
The challenges were scored with a common understanding rubric, and the authors recorded each agent's complete interactions with IDA Pro. That let them compare how humans and agents investigate a binary as well as what they recover. They also ran ablations on code representation, and measured context-window saturation across 4,000 real-world binaries.
Results
GPT-5.5 Pro, Claude Opus 4.8 and Kimi K3 were the best agents. Weaker models stayed far behind. Open-weight models improved a lot with richer code representations, while frontier models showed no significant change. Humans revisit functions and follow local structure. Strong agents jump across the binary, fetch information broadly and rarely return, a behaviour the authors call greedy fetching. Large windows hold many programs, but larger applications and lower-level representations quickly exhaust them. The paper concludes that scaling will need better navigation, selective retrieval and persistent analysis state. It was submitted on 10 October 2026.