The authors built TeamBench, a benchmark that forces multi-agent systems to actually coordinate by using operating-system-level access controls, not just prompts. They split tasks across Planner (reads requirements), Executor (edits workspace), and Verifier (certifies output) roles where no single agent can see everything, edit code, and approve the result. When roles are only separated by prompts, teams reach the same pass rate as sandbox-enforced teams, but verifiers try to do the executor's job 3.6× more often, and verifiers approve wrong answers 49% of the time. A human study shows that role separation reveals coordination patterns (solo humans work through directly, human-agent pairs collapse into quick approval, human teams coordinate more carefully).
Main takeaways:
- Pass rate alone masks whether agents actually coordinated or one agent just did everything; role separation makes this visible
- Prompt-only vs. sandbox-enforced teams reach similar pass rates, but prompt-only produces 3.6× more role violations (e.g., verifier trying to edit code)
- Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier actually improves partial scores
- Teams help when single agents struggle but hurt when single agents already perform well (team value is conditional)
- 40-session human study shows role separation exposes coordination patterns that aggregate metrics miss