How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

arXiv:2510. 01163v2 Announce Type: replace-cross Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising effectiveness, enabling models to adapt to new tasks from only a handful of examples.