The authors build large-scale infrastructure for systematically generating, categorizing, and evaluating jailbreak prompts. They compose 114,000 adversarial prompts by applying 912 strategies to 125 harmful seeds, labeling each with one of 14 cybersecurity attack categories (malware, phishing, etc.) and ranking strategies by category-specific effectiveness. They then fine-tune models to generate fluent jailbreaks on demand (no templates, no gradient search) and introduce OPTIMUS, a continuous evaluator that jointly scores semantic similarity to the harmful seed and harmfulness probability, revealing a "stealth-optimal" regime that binary attack-success-rate misses.
Main takeaways:
- Dataset of 114,000 jailbreak prompts categorized by adversarial intent (malware, phishing, privilege escalation, etc.) with strategies ranked by effectiveness per category.
- Fine-tuned generators produce fluent jailbreaks at inference time with perplexity 24–39 (vs. 40–140 for AutoDAN/AmpleGCG) and safety-filter evasion rates 0.29–0.51.
- OPTIMUS is a training-free continuous metric balancing semantic similarity to the seed and harmfulness, generalizing across strategies without task-specific tuning.
- Exposes a stealth-optimal operating point (similarity ≈0.57, harmfulness ≈0.43) that binary attack-success-rate metrics miss entirely.