The authors argue that agent benchmarks reporting only pass/fail scores hide critical validity problems—shortcuts, scaffold limitations, and dangerous actions—and that systematic log analysis (tracking inputs, execution steps, and outputs) is necessary for credible evaluation. They present a taxonomy of validity threats uncovered by log analysis, develop guiding principles, and demonstrate on tau-Bench Airline that pass@5 was under-elicited by nearly 50% and that outcome metrics missed deployment failure modes visible only in execution logs.
Main takeaways:
- Pass/fail scores alone can inflate/deflate capability via shortcuts, miss real-world failure modes, and conceal dangerous actions
- Log analysis—systematic tracking of agent inputs, steps, and outputs—surfaces these hidden validity threats
- On tau-Bench Airline, log analysis revealed pass@5 was under-elicited by ~50%
- Execution logs exposed deployment failure modes invisible to outcome metrics
- Provides a taxonomy of threats and guiding principles for log analysis, with recommendations for benchmark creators, model developers, evaluators, and deployers