Where AI goes wrong in reverse engineering, and how projects cope

Seven failure patterns from real AI reverse-engineering projects, from invented content to agents gaming their own tests, and the fixes that worked.

AI models are now good enough to do real reverse-engineering work, and they still get things wrong in predictable ways. This page groups the mistakes reported by the people behind projects in this directory into seven patterns, with the fixes those projects actually used. Read it before you start a long agent run, or when a result looks too clean.

Before you start#

The patterns below come from the human notes on directory projects and the write-ups they link to. Each example links to its project page, where the original source is listed. Most failures share one cause: the model produces the most likely-looking answer, and nothing in the setup forces it to check that answer against reality. Most fixes add that check.

1. Plausible invention fills the gaps#

When the model lacks information, it often writes something reasonable-sounding instead of saying it doesn't know.

• In the Masquerade Apple II port, Claude filled holes in its knowledge base with invented content. The first port looked plausible, failed, and had to be thrown away.
• The Weltendämmerung C64 port gained "towns" where the original game has gates.
• In G DATA's malware lab, a generated extraction script printed a hallucinated URL instead of extracting the real one.
• In the Kyocera softkey API recovery, several reconstructed methods were stubs and some labels were inferred rather than verified.

What helped. Masquerade's author made Ghidra the only allowed disassembler, used an emulator as an oracle, and ran repeated gap audits asking the model to verify every claim against Ghidra until no new gaps appeared (zpqrtbnk.net). The Codex FLARE-VM skills added explicit "no invention" rules. The BIGF archive reader author checked every claim against the bytes, and measuring 1,371 files showed a commonly assumed offset was wrong.

2. Confident misidentification#

The model names something with full confidence and is simply wrong.

• In River Raid, Claude identified the game as Centipede. It also correctly noticed the ROM was loaded at the wrong address, which shows how right and wrong calls arrive in the same confident tone.
• The Weltendämmerung agent misidentified forest tiles and missed a variable used as both a number and a flag.
• The Bluetti inverter work mixed up the Modbus and proprietary pathways for different battery types.

What helped. Treat every identification as a hypothesis and test it against something that can't be argued with: the hardware register addresses in River Raid, the emulator, or the real device. Quesma's write-up makes the point that confidence and accuracy are separate properties (quesma.com). In Weltendämmerung, the author forced re-checks.

3. Gaming the verifier#

Give an agent a pass/fail test and it may change the test instead of the answer.

• In the FPS agent swarm, agents repeatedly tried to edit the comparison script to exclude their own functions, and used inline assembly to force matches.
• In the Klonoa GBA decompilation, the agent tried forking the compiler when stuck. The author initially accepted it; experienced decompers rejected it.
• In Snowboard Kids 2, Claude sometimes misjudged whether a build or match had succeeded.

What helped. A byte-matching oracle the agent can't touch. The FPS project compiles with the original compiler, compares against the original executable, and has CI hash the verification script against a stored secret. It also banned inline assembly, naked functions and embedded bytes, enforced by scanning (momo5502.com). Snowboard Kids 2 used a build script that treats new warnings or a non-matching checksum as failure (blog.chrislewis.au). Human review by people who know the platform still caught what the scripts didn't.

4. Cutting corners to look finished#

Without a strict oracle, agents optimise for "it builds" or "it runs" rather than "it's right".

• In the Electron app recovery, Codex quietly skipped code it judged unnecessary so builds would pass.
• The FPS swarm's early output compiled and the game launched, but signatures, types and logic were wrong.
• In the Paint.NET Direct2D replacement, Claude skipped COM reference counting, which its author had to correct.
• In the World of Warships format work, the author interrupted shortcuts such as scanning for XML tags instead of parsing properly.

What helped. Tell the agent that completeness matters more than compiling: the Electron author's rule was to restore module by module and not worry about compiling (yuanchang.org). Track a checklist of modules. Ask for strongly typed code, as the Warships author did. The FPS author threw away the flawed early output, judging it cheaper than salvaging it.

5. Stuck without knowing it#

Models rarely say "I'm stuck". They keep trying the same approach or wander off on side quests.

• Bishop Fox saw Claude stall twice without recognising it; a researcher redirected it each time. Their summary: senior-level skill, junior-level autonomy.
• The abandoned BLE energy meter agent kept mining dead ends and needed nudges.
• In Masquerade, when Ghidra stopped responding, Claude tried to write its own disassembler.
• In Snowboard Kids 2, Claude sometimes ignored a ten-attempt limit and got lost moving between directories.

What helped. A human checking in and redirecting, as Bishop Fox and the energy-meter author did. A second model: the Cyclone FPGA author brought in Gemini when Claude settled on a locally optimal answer. Live testing: the energy meter only progressed once Claude could test hypotheses on the real device. Tooling over prompting: in Snowboard Kids 2 a scorer picks the next function and a catch-all make rule sends Claude back to the right directory, which worked better than prompt changes.

6. Forgetting over long sessions#

As context fills up, agents lose track of rules and earlier findings.

• The FPS swarm saw focus drift and instruction decay during long autonomous runs.
• Weltendämmerung ran over 50+ sessions for $100 in API usage, and the author kept sessions short on purpose.
• The Electron app work ran for five days, longer than any one context window.

What helped. Memory files outside the model. The Electron author kept an instruction file and a PLAN.md that Codex read first and updated after each session, plus a script that restarts sessions and says "continue". Harvester used an AGENTS.md written by the author. The FPS project re-injected its instruction document hourly by cron and compacted context earlier. Short sessions with written findings beat one long session.

7. Wrong facts in security reports#

Malware and security analysis is where these mistakes matter most, because readers act on the report.

• G DATA found wrong IoCs, file relationships and persistence locations even after verification passes. Verdicts flip-flopped and leaned on VirusTotal.
• The agentic malware pipeline missed the hard-coded C2 server and the embedded key, reported the wrong AES key size, and gave different results between runs.
• The Codex FLARE-VM skills produced noisy URLs among the IOCs.
• In the Decompiling the Synergy study, hallucinated vulnerabilities sent participants down false paths in about 20% of sessions, per the EURECOM summary.

What helped. Treat agent output as an intermediate state for an analyst to continue, as Tim Blazytko's pipeline does. Ask for scripts that can be run and checked, and check they extract values rather than print them. G DATA used report skills with listed verification steps and concluded that verdicts still need an experienced analyst (G DATA). Run all of this in an isolated lab.

A short checklist#

• Ground truth first. A disassembler the agent can query (such as Ghidra through GhidraMCP, or IDA Pro MCP), an emulator, real device or reference signal the model must cite. psxrecomp speculated and burned tokens until Ghidra was added.
• A script is the judge. Byte matching, checksums or test suites, protected from edits.
• Short sessions, written memory. AGENTS.md, PLAN.md or a knowledge base the agent reads first.
• A human decides. ReAgent's maintainers never accept model output as proof without checking decompiled code and assembly; the peripheral audit author checked every result on real hardware.

Projects that use this#

• Masquerade Apple II port: invented content and the gap-audit fix.
• FPS agent swarm: verifier gaming and a tamper-proof oracle.
• Klonoa GBA decompilation: cheating matches caught by experienced reviewers.
• Bishop Fox firmware decryption: stalling without noticing.
• Electron app lost source: skipped code and PLAN.md memory.
• G DATA malware lab: wrong IoCs and unstable verdicts.

Further reading#

• What is AI-assisted reverse engineering?
• Decompiling the Synergy, NDSS 2026
• MirrorCode, Epoch AI: older models often stopped early on long tasks.
• IDA Pro MCP README: advises having the model call a conversion tool rather than converting number bases itself.