The authors show that standard self-play red-teaming—where the same model plays both attacker and defender—has fundamental flaws: it can converge to useless equilibria (like always refusing) and collapses into self-consistency rather than maintaining adversarial pressure. They propose Anchored Bipolicy Self-Play, which trains separate LoRA adapters for attacker and defender roles on top of a frozen base model, achieving 100× better parameter efficiency than full fine-tuning while maintaining real adversarial pressure. Testing on Qwen2.5 models shows improved safety without hurting reasoning.
Main takeaways:
- Self-play with a single model doesn't create real adversarial dynamics—attacker and defender just learn to be consistent with each other
- Separating attacker/defender into distinct LoRA adapters (frozen base) keeps the optimization stable but preserves adversarial pressure
- 100× more parameter-efficient than full fine-tuning, with better safety scores on standard benchmarks
- Cross-play experiments confirm that the resulting attacker and defender models are individually stronger than self-play versions