The authors design DisagMoE, a system for training mixture-of-experts (MoE) LLMs more efficiently by separating attention and expert (feed-forward) layers onto different GPU groups and pipelining their communication. Standard expert-parallelism training hits network bottlenecks because all-to-all communication (routing tokens to different experts across nodes) doesn't overlap well with computation. DisagMoE disaggregates the model so attention and FFN layers run on disjoint hardware, uses a multi-stage pipeline with unidirectional communication, and balances computation-to-communication ratios across the groups. On 16-node H800 clusters, DisagMoE delivers up to 1.8× training speedup.
Main takeaways:
- MoE models scale by activating only a subset of experts per token, but training them with expert parallelism causes severe all-to-all communication stalls.
- Prior overlap techniques can't fully hide communication because attention and FFN layers have different computation-to-communication ratios.
- DisagMoE splits attention and FFN onto separate GPU groups, uses a unidirectional many-to-many pipeline, and applies a roofline model to allocate bandwidth optimally.
- Implemented in Megatron-LM, DisagMoE achieves up to 1.8× speedup on multi-node clusters.
- The method is purely about distributed training infrastructure, not model behavior or alignment.