CREBench
A benchmark framework for evaluating Large Language Models in cryptographic binary reverse engineering across 432 challenges and four difficulty levels.
0 projects
About CREBench
CREBench is a benchmark and evaluation framework designed to assess the capabilities of Large Language Models (LLMs) in cryptographic binary reverse engineering. It features a corpus of 432 challenges derived from 48 algorithms, utilizing three key modes (hardcode_plain, fragmented_build, and weak_prng_seeded) and three difficulty levels (O0, O3, and constxor). The evaluation is structured into a four-level ladder: algorithm identification (L1), key/IV extraction (L2), wrapper-level code reimplementation (L3), and flag recovery (L4). The system provides an agent runtime, a sandbox environment, and specialized tools for pass@k evaluation. It supports a variety of models including GPT-5.4, Claude-Sonnet-4-6, and Gemini-2.5-Pro. Setup requires Python 3.10+, Docker, and a specific version of Ghidra (11.0.1) for the runtime sandbox. Users can perform full matrix sweeps or single-run debugging using provided Python scripts. For token accounting, it integrates with LiteLLM, though documentation warns against specific vulnerable versions of that library.
Summarised from the official page linked above.