DualPipe
An innovative bidirectional pipeline parallel scheduling algorithm designed for large-scale distributed training of giant MoE architectures. By running forward and backward micro-batches from both ends of the pipeline simultaneously, it reduces idle bubble time to less than 10%.
https://github.com/deepseek-ai/DualPipe
- Drastically shrinks the dreaded 1F1B pipeline bubble down to negligible overhead.
- Interleaves forward and backward passes across dual computation directions without deadlocks.
- Open-source week described DualPipe as bidirectional pipeline parallelism for V3/R1 training. V4-Pro's model card lists 1.6T total parameters.
Plain-language overview
DualPipe is a bidirectional pipeline-parallel schedule for very large models. The DualPipe repository explains how forward and backward micro-batches enter from both ends of the pipeline stage graph so idle "bubble" time drops compared with classic 1F1B schedules — DeepSeek's OpenSource Week materials cite sub-10% bubble overhead for MoE training.
Pipeline parallelism splits layers across devices; without careful scheduling, devices wait on neighbors. DualPipe is aimed at teams training hundred-billion- to trillion-parameter MoEs who already shard experts with DeepEP and store checkpoints on 3FS. It is not an API feature you toggle in deepseek-flash requests.
Read the README's diagrams and pseudocode before adopting DualPipe in your trainer; integration assumes PyTorch distributed primitives and compatible checkpoint formats. For product-facing context, see the release timeline when DeepSeek announced V3/R1 training stacks.