The authors analyzed a multi-agent orchestration competition where teams built industrial automation systems. They looked at submission logs, leaderboards, and source code to understand what the evaluation actually measured and what worked. The public leaderboard hit a ceiling that better prompts couldn't break through, and scores on the hidden test set told a completely different story than public scores—especially for execution tasks where the correlation was actually negative.
Main takeaways:
- Public leaderboard scores maxed out at 72.73% and more sophisticated prompts didn't push past that ceiling
- Hidden test scores barely correlated with public scores in planning (r=0.69) and were negatively correlated in execution (r=-0.13), meaning some low-ranked teams did best on held-out data
- A bug in the scoring formula made one term contribute almost nothing (0.05 points max), and fixing it would have swapped the top two winners
- Winning execution systems mostly added guardrails—better response filtering, fallback logic, and context control—not fancy new agent architectures
- 149 registered teams collapsed to only 11 with complete scores, showing huge drop-off between signup and actual participation