DualPipe

Checked on 15 Nov 2026 against DualPipe.

An innovative bidirectional pipeline parallel scheduling algorithm designed for large-scale distributed training of giant MoE architectures. By running forward and backward micro-batches from both ends of the pipeline simultaneously, it reduces idle bubble time to less than 10%.

https://github.com/deepseek-ai/DualPipe

  • Drastically shrinks the dreaded 1F1B pipeline bubble down to negligible overhead.
  • Interleaves forward and backward passes across dual computation directions without deadlocks.
  • Open-source week described DualPipe as bidirectional pipeline parallelism for V3/R1 training. V4-Pro's model card lists 1.6T total parameters.

Plain-language overview

DualPipe is a bidirectional pipeline-parallel schedule for very large models. The DualPipe repository explains how forward and backward micro-batches enter from both ends of the pipeline stage graph so idle "bubble" time drops compared with classic 1F1B schedules — DeepSeek's OpenSource Week materials cite sub-10% bubble overhead for MoE training.

Pipeline parallelism splits layers across devices; without careful scheduling, devices wait on neighbors. DualPipe is aimed at teams training hundred-billion- to trillion-parameter MoEs who already shard experts with DeepEP and store checkpoints on 3FS. It is not an API feature you toggle in deepseek-flash requests.

Read the README's diagrams and pseudocode before adopting DualPipe in your trainer; integration assumes PyTorch distributed primitives and compatible checkpoint formats. For product-facing context, see the release timeline when DeepSeek announced V3/R1 training stacks.

All libraries