The authors test whether LLMs can adjust their answers to match how certain the retrieved context is—e.g., hedging when told a fact is "uncertain" or being confident when it's "certain." They find systematic failures across eight models: LLMs forget their own prior knowledge after seeing uncertain context, misinterpret certainty cues, and overtrust complex contexts. To fix this without retraining, they propose a three-part interaction strategy: remind the model of its prior knowledge, explicitly recalibrate the certainty level in the prompt, and simplify the context. This reduces "obedience errors" (mismatches between context certainty and response confidence) by 25% on average.
Main takeaways:
- LLMs struggle to recall prior knowledge after observing an uncertain context, even when the context explicitly says "this might be wrong."
- They misinterpret expressed certainties—e.g., treating "possibly" the same as "definitely."
- Complex or verbose contexts are overtrusted regardless of stated certainty level.
- A prompt-engineering fix (prior reminder + certainty recalibration + simplification) cuts obedience errors by ~25% without model changes.
- The evaluation metric, "context-certainty obedience," measures how well response confidence tracks stated context certainty, which matters in high-stakes domains like medicine and finance.