The paper challenges the assumption that chain-of-thought reasoning reduces shallow biases. Testing position bias (models favoring option A or D in multiple-choice questions), the authors find the opposite: longer reasoning trajectories correlate with stronger position bias within any reasoning-capable model. Across thirteen reasoning configurations (R1-distilled models, base models with CoT prompting, DeepSeek-R1 at 671B) on MMLU, ARC-Challenge, and GPQA, twelve show positive partial correlation between trajectory length and position bias (0.11–0.41, all p<0.05). A truncation experiment confirms causality: resuming reasoning from later points increases the shift toward position-preferred options. At 671B, aggregate bias is near zero, but the longest-trajectory quartile still shows the effect, suggesting accuracy gates the expression of length-driven bias rather than removing the mechanism.
Main takeaways:
- More reasoning (longer trajectories) → more position bias, not less, within reasoning-capable models.
- Twelve of thirteen reasoning configurations show significant positive correlation between trajectory length and position bias (0.11–0.41).
- Truncation experiment: resuming reasoning from later points causally increases position-biased shifts (16%–32% for R1-Qwen-7B).
- At 671B scale, aggregate bias is low (0.019) but the longest quartile still shows the effect (0.071)—accuracy suppresses but doesn't eliminate the mechanism.
- CoT reasoning replaces baseline direct-answer bias with a new length-accumulated bias; reasoning models should not be treated as order-robust in MCQ evaluation.