The authors built Agentick, a benchmark designed to compare very different kinds of AI agents (from-scratch reinforcement learning, large language models, vision-language models, hybrids, and even humans) on the same set of sequential decision-making tasks. They tested 27 agent configurations across 37 procedurally generated tasks spanning planning, multi-agent coordination, exploration, and other capabilities, finding that no single approach wins everywhere—GPT-5 mini leads overall but RL methods dominate on planning tasks, and a reasoning harness boosts LLM performance 3–10×.
Main takeaways:
- No single agent type dominates: GPT-5 mini scored highest overall (0.309 normalized), but PPO (a reinforcement learning algorithm) beat it on planning and multi-agent tasks.
- Adding a reasoning harness (structured prompting framework) multiplied LLM performance by 3–10× depending on the task.
- ASCII text observations consistently outperformed natural-language descriptions for all agent types.
- All approaches have huge headroom—even the best agents are far from oracle performance, showing plenty of room for improvement.
- The benchmark ships with reference policies, pre-built fine-tuning datasets, and a live leaderboard to enable reproducible comparisons.