The authors show that fine-tuning APIs can be jailbroken using Direct Preference Optimization (DPO) with as few as 10 completely benign training examples. Each example pairs a harmless question with a helpful answer (preferred) and a refusal (dispreferred)—training data that looks identical to what a legitimate user might submit to reduce over-refusal. Because DPO teaches the model to prefer helpful responses over refusals in general, this objective transfers to harmful prompts, achieving attack success rates of 59-82% on GPT-4o variants at costs under $2.
Main takeaways:
- DPO fine-tuning creates a stronger, harder-to-detect jailbreak vector than supervised fine-tuning because the training data itself is indistinguishable from legitimate use.
- Only 10 harmless preference pairs (the minimum OpenAI accepts) are enough to broadly suppress refusal behavior across harmful prompts outside the training set.
- The attack works because DPO directly optimizes against refusals as a class, not just on the specific prompts in the training data.
- Open-weight models show the effect can emerge from a single benign preference pair when no minimum data requirement is enforced.
- The method works across GPT-4o, GPT-4.1, and their smaller variants, with success rates between 54% and 82%.