The authors show that instructions in language models primarily affect how output tokens are produced rather than how input tokens are processed. Using layer-by-layer probing across five tasks, they find that task-specific information in the input stays mostly stable regardless of instructions, while the same information in output tokens varies dramatically and tracks actual behavior. Attention interventions confirm this causally: blocking instruction influence to all tokens hurts performance, but blocking it only to input tokens barely matters.
Main takeaways:
- Instructions shape what happens during output generation, not during input understanding—an asymmetry between "processing" and "production" stages
- Task information encoded in output tokens correlates strongly with behavior; the same information in input tokens correlates weakly
- Blocking instruction attention flow to output tokens tanks performance; blocking it only to input tokens has minimal effect
- The effect becomes sharper with model scale and instruction-tuning, both of which disproportionately amplify the production stage
- Suggests measuring model internals by token position (input vs output) reveals more than treating all positions the same