The authors investigate whether frontier LLMs can automate the evaluation of AI agents—tasks that involve multi-step reasoning and tool use. They find that simply prompting coding assistants fails badly (30% execution success, over-engineered metrics), so they build EvalAgent, a system that uses "evaluation skills" (procedural instructions, reusable code templates, and API docs) to generate complete evaluation pipelines. They introduce a meta-evaluation benchmark (AgentEvalBench, 20 agents) and an Eval@1 metric measuring whether generated evaluation code runs correctly on the first try.
Main takeaways:
- Strong coding ability doesn't automatically make LLMs good at building agent evaluations—raw prompting yields only 30% working evaluations.
- EvalAgent packages domain expertise as "evaluation skills" (instructions, code templates, retrieved docs) and raises Eval@1 from 17.5% to 65%.
- Human experts prefer EvalAgent's evaluations 79.5% of the time over baseline approaches.
- Removing the evaluation skills drops Eval@1 back to 30%, showing they're critical for complex evaluation tasks.
- The system produces focused evaluations instead of metrics bloat (12+ metrics per agent in naive approaches).