ExploitGym is a benchmark testing whether AI agents can turn known security vulnerabilities into working exploits—taking a program input that triggers a bug and progressively extending it into code that achieves unauthorized access or execution. The benchmark contains 898 real-world vulnerabilities across userspace programs, V8 JavaScript engine, and Linux kernel, with varying security protections, and finds that frontier models (Claude Mythos Preview and GPT-5.5) successfully exploit 157 and 120 instances respectively.
Main takeaways:
- Exploitation requires low-level reasoning about memory layout, runtime adaptation, and sustained progress over long horizons—it's harder than just detecting vulnerabilities
- Even with widely used security defenses enabled, models retain non-trivial success rates at producing working exploits
- The benchmark packages all configurations in reproducible containerized environments and varies protections to isolate their impact on agent performance
- This represents an under-evaluated but critical capability with dual-use implications—it supports defensive security workflows but also lowers the barrier for offense