Multi-turn jailbreaks gradually steer a conversation toward an unsafe answer rather than stating a harmful request upfront, but existing methods are evaluated with different budgets, judges, retry rules, and strategy generation, making it unclear whether gains reflect stronger attacks or different conditions. MT-JailBench is a modular benchmark that implements each attack as five interacting modules (evaluation function, attack strategy, prompt generation, prompt refinement, flow control) so methods can be compared under fixed conditions. Component-wise analysis shows that resource budgets and evaluation functions are major confounders; prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. Recomposing the best components yields a strong configuration that outperforms source attacks and generalizes across LLMs.
Main takeaways:
- Multi-turn jailbreaks accumulate conversational context to reach unsafe answers, but existing evaluations differ in budgets, judges, retries, and strategy generation, confounding comparisons.
- MT-JailBench decomposes each attack into five modules (evaluation, strategy, prompt generation, refinement, flow control) for fair comparison and component-level analysis.
- Resource budgets (turns, retries, interactions, sampled strategies) and evaluation functions are major confounders; controlling them changes the ranking of attacks.
- Prompt generation accounts for most performance variation; refinement and flow control provide moderate gains; explicit dynamic strategy generation isn't always necessary.
- Recomposing the best components yields a configuration that outperforms source attacks and generalizes across diverse LLMs.