The authors show that standard parameter sharing in multi-agent reinforcement learning fails when agents have symmetric observations—everyone gets identical action distributions, making role differentiation impossible. They propose Diamond Attention, where each agent samples a random number that creates a temporary rank ordering; lower-ranked agents can't see higher-ranked ones in agent-to-agent attention, breaking symmetry and enabling coordination. This implements a random-bit coordination protocol in one communication round.
Main takeaways:
- Under symmetric observations, a shared deterministic policy outputs identical actions for all agents, preventing any role specialization (e.g., in the XOR game, deterministic baselines plateau at 0.5 success; Diamond Attention hits 1.0).
- Each agent samples a scalar random number per timestep to induce a transient rank; lower-ranked agents' attention to higher-ranked peers is masked.
- Set-based attention enables zero-shot generalization: a policy trained on 4 agents works on 2–8 agents without retraining.
- Replacing structured randomness with standard dropout gives 0% win rate, confirming that protocol structure (not just noise) is essential.
- Demonstrates that randomness is sometimes theoretically necessary for coordination, not just an engineering trick.