The authors study when agents should use accumulated experience during decision-making, not just how experience should be stored or updated. They introduce ExpWeaver, which exposes experience as an optional resource during reasoning so agents can invoke it only when needed, rather than injecting it at every step or only at initialization. Across four agent frameworks, seven LLM backbones, and three environments, ExpWeaver consistently outperforms fixed usage strategies. Analysis shows agents learn to invoke experience selectively at beneficial decision points and under higher reasoning uncertainty.
Main takeaways:
- Most self-evolving agents use rigid experience-injection strategies (always at start, or always at every step), ignoring whether experience is actually needed
- ExpWeaver makes experience optional during reasoning, letting the agent decide when to retrieve and use it
- Consistently achieves best performance across diverse frameworks, model sizes, and task types
- RL training amplifies the selective-invocation behavior
- Usage-pattern and entropy analyses show agents invoke experience when facing higher decision uncertainty and at beneficial choice points
- Suggests experience utilization (when to use stored knowledge) is as important as experience construction (what to store)