The authors resolve contradictions in the literature about whether LLMs follow provided documents or stick to their trained knowledge when they conflict. They propose a three-regime framework: Regime 1 is single-source updating (models follow documents when evidence is coherent), Regime 2 is competitive integration (models rely on parametric certainty when choosing between conflicting sources), and Regime 3 is task-appropriate selection (task framing determines whether models use context or parameters). They validate this across five frontier models, showing parametric certainty predicts behavior in Regime 2 and task framing flips context-following rates from near-100% to 6-71% in Regime 3.
Main takeaways:
- Prior studies found contradictory results (models ignoring documents ~50% vs. following them ~96%) because they unknowingly studied different regimes
- Regime 1: single-source updating—models follow documents when evidence is coherent and there's no strong parametric knowledge
- Regime 2: competitive integration—when parametric knowledge competes with context, parametric certainty (not just strength) predicts whether models override the document
- Regime 3: task-appropriate selection—task framing ("use your knowledge" vs. "use the document") flips context-following from nearly 100% to single digits
- Parametric strength (training frequency) and parametric uniqueness (encoding consistency) are orthogonal dimensions; strength is the operative predictor in factual domains