The authors built two LLM-based systems (RADAGAS and RefleXQLi) that automatically generate adversarial SQL injection attacks to test web application firewalls (WAFs). They ran 240 experiments producing 240,000 attack payloads and tested them against 10 different WAFs including rule-based, AI/ML-based, and commercial systems using GPT-4o, Claude, and DeepSeek as the attack generators.
Main takeaways:
- RADAGAS-GPT4o achieved a 22.73% bypass rate overall and was especially effective against AI/ML-based WAFs (92% on WAF-Brain, 80% on CNN-WAF) but struggled with rule-based WAFs (0-6% on ModSecurity/Coraza)
- Less diverse payload generation actually achieved more bypasses, though this strategy fails completely if the initial payload doesn't work
- LLM-generated attacks can automatically probe defense mechanisms at scale, turning adversarial testing into a largely automated process
- Commercial WAFs like AWS and Cloudflare were tested but specific bypass rates weren't highlighted in the abstract