DualPipe
An innovative bidirectional pipeline parallel scheduling algorithm designed for large-scale distributed training of giant MoE architectures. By running forward and backward micro-batches from both ends of the pipeline simultaneously, it reduces idle bubble time to less than 10%.
https://github.com/deepseek-ai/DualPipe
- Drastically shrinks the dreaded 1F1B pipeline bubble down to negligible overhead.
- Interleaves forward and backward passes across dual computation directions without deadlocks.
- Open-source week described DualPipe as bidirectional pipeline parallelism for V3/R1 training. A 1.7T scaling claim was removed; V4-Pro's card lists 1.6T.