The authors show that frontier LLMs still produce verifiable security vulnerabilities 23% of the time even when prompted to write secure code, then introduce SecureForge: an automated pipeline that finds benign prompts that trigger these bugs, amplifies them into a large synthetic corpus using Markovian sampling (to keep diversity and error rates realistic), and then iteratively optimizes system prompts to reduce output vulnerabilities. The resulting system prompts cut vulnerabilities by up to 48% without hurting unit-test pass rates, and transfer zero-shot to real user prompts despite never seeing them during optimization.
Main takeaways:
- Frontier models produce statically detectable security vulnerabilities in 23% of 250 benign coding prompts, even when asked to write secure production code
- SecureForge identifies trigger prompts, expands them via Markovian sampling to maintain diversity and error rates, then optimizes system prompts to reduce vulnerabilities
- Achieves a Pareto improvement: up to 48% fewer vulnerabilities while maintaining or improving unit test success
- The optimized system prompts transfer zero-shot to in-the-wild coding agent prompts without exposure to real user distributions
- Demonstrates that prompt-level defenses can meaningfully reduce security risks in LLM code generation