The authors propose ξ-DPO, a variant of preference optimization (fine-tuning LLMs from human preference data) that simplifies hyperparameter tuning. Existing methods like SimPO have two coupled hyperparameters (β and γ) that are hard to tune jointly because the margin γ doesn't have a consistent interpretation across datasets with different reward distributions. ξ-DPO reformulates the objective to use a "ratio reward margin" — the ratio of chosen to rejected response probabilities — which is bounded and interpretable, and can be set directly from the initial reward gap distribution without trial-and-error.
Main takeaways:
- SimPO's margin hyperparameter γ is not easily interpretable across datasets because it depends on the reward gap structure; tuning β and γ jointly is difficult.
- ξ-DPO redefines the reward as a ratio (chosen/rejected) rather than a difference, yielding a bounded margin ξ that explicitly represents desired relative separation.
- The ratio formulation cancels the effect of β, eliminating one hyperparameter from the tuning problem.
- ξ can be determined from the initial reward gap distribution, avoiding repeated trial-and-error tuning across datasets.
- Reformulates the preference objective as minimizing distance between reward gaps and optimal margins rather than maximizing likelihood of reward gaps.