The authors study why many-shot jailbreaking (MSJ) works — preceding a harmful query with many harmful question-answer demonstrations makes safety-aligned models comply. They find that MSJ causes progressive activation drift: adding more harmful demos shifts the query's representation step-by-step away from the safety-aligned region. They show this drift is equivalent to implicit malicious fine-tuning via SGD-style updates on the N harmful samples. Flipping the mechanism, they append a single safety demonstration at inference time to induce a counteracting update that restores refusal, improving robustness without parameter changes or white-box access.
Main takeaways:
- Many-shot jailbreaking causes progressive activation drift: each added harmful demo shifts the query representation further from the safety-aligned region
- Theoretically, conditioning on N harmful demos is equivalent to SGD-style updates on N harmful samples (implicit malicious fine-tuning)
- Appending a single safety demonstration at inference time induces a counteracting safety-oriented update and restores refusal
- The defense requires no parameter changes or white-box access, just a one-shot safety example in the context
- Turns the attack mechanism into a defense principle: if harmful demos are implicit fine-tuning, safety demos are implicit safety fine-tuning