The authors built OLIVIA, a system that lets ReAct agents (LLMs that reason, act, observe in a loop) improve their action choices during deployment through online learning. The key idea is modeling the final action-selection step as a contextual bandit: the system scores candidate actions using the LLM's frozen hidden states as context, maintains uncertainty estimates, and updates online from action-level feedback using upper-confidence-bound exploration. This adapts behavior directly at the action interface without retraining the LLM or changing its reasoning process. On four benchmarks, OLIVIA consistently beats static ReAct and prompt-based adaptation with minimal compute overhead.
Main takeaways:
- Most inference-time adaptation for agents uses prompting or retrieval, which influences behavior indirectly through context—OLIVIA instead adapts the final action-selection layer directly.
- The action layer is modeled as a lightweight linear bandit over candidate actions, with the LLM's frozen hidden states as decision features.
- This design gives explicit uncertainty estimates, lets you update from action-level feedback online, and preserves the underlying reasoning process.
- Upper-confidence-bound exploration makes the policy improve sample-efficiently with low computational cost.
- Consistently outperforms static ReAct and prompt-based baselines across benchmarks, showing that explicit online decision layers are a viable alternative to pure prompting.