Most jailbreak papers report attack success on one or a few hand-picked parameter configurations, but jailbreaks expose many tunable knobs (prompt templates, conversation rounds, cipher dispersion, etc.) and performance varies hugely across them. The authors argue single-configuration ASR is insufficient and propose two new metrics: Variant Sensitivity Measure (how far the best ASR deviates from the mean across variants) and Union Coverage (fraction of prompts that jailbreak under any tested configuration). Empirically, they show that for PAIR the best template reaches 69% ASR on Mistral-7B but union coverage hits 88%, and for bijection the best variant gets 81% but the union covers 100% of HarmBench-100.
Main takeaways:
- Jailbreak attacks have many configurable parameters; reporting only the best configuration hides how typical that performance is and how much of the attack surface single-variant evaluation misses.
- Variant Sensitivity Measure (VSM) quantifies how much the best ASR deviates from the mean across the tested variant space.
- Union Coverage (UC) measures the total fraction of prompts that jailbreak under any tested configuration, capturing the full threat surface.
- Empirical examples: PAIR best-template ASR 69% on Mistral-7B, but UC 88%; bijection best-variant 81%, UC 100%. Single-config ASR drastically underestimates threat.