DeepSeek Research Hub

Open-source libraries

DeepSeek OpenSource Infrastructure

During DeepSeek's OpenSource Week, the engineering team open-sourced the entire distributed high-performance systems stack powering their frontier clusters. These repositories provide the extreme compute, communication, and memory efficiency required for modern MoE architectures.

github.com/deepseek-ai
Attention & Kernels

FlashMLA

Hopper TMA-Accelerated Multi-Head Latent Attention Kernel

An ultra-fast decoding kernel tailored specifically for Multi-Head Latent Attention (MLA) on NVIDIA Hopper architectures. Employs Tensor Memory Accelerator (TMA) asynchronous memory copies to maximize memory bandwidth utilization during variable-length generation.

Hopper TMAMLA AttentionKV CompressionCUDA C++High Throughput
✓Optimized for variable sequence length decoding on H800/H100/H200 GPUs.
✓Takes full advantage of Hopper TMA hardware features for zero-copy global-to-shared memory transfers.
✓The repository describes optimized MLA kernels. A 300–355 tok/s claim for V4.1-Flash was removed because it is not on the model card.
git clone https://github.com/deepseek-ai/FlashMLA.git
Repo
Communication & MoE

DeepEP

Ultra-Low Latency MoE NVLink & InfiniBand All-to-All Communication

The first open-source communication library purpose-built for Mixture-of-Experts (MoE) dispatch and combine operations. Implements custom NVLink kernels within nodes and RDMA InfiniBand across nodes, overlapping MoE communication directly with GPU matrix computation.

MoE All-to-AllNVLink CustomRDMA InfiniBandZero BubbleNCCL Alternative
✓Eliminates standard NCCL all-to-all serialization bottlenecks across thousands of GPUs.
✓Overlaps expert token dispatch and combine with upstream layer tensor calculations.
✓Critical communication foundation for both V3, V4-Pro, and V4.1-Flash clusters.
git clone https://github.com/deepseek-ai/DeepEP.git
Repo
Compute & GEMM

DeepGEMM

Clean, High-Efficiency FP8 General Matrix Multiplication with Tile Scaling

A clean, performant FP8 GEMM library built specifically for Hopper Tensor Cores. Supports both standard continuous GEMM and fine-grained grouped MoE GEMMs with fine-grained per-tile scaling, achieving over 95% of theoretical peak FP8 tensor core throughput.

FP8 GEMMHopper Tensor CoresTile ScalingCUTLASSQuantization
✓Implements fine-grained 128x128 tile scaling to prevent FP8 underflow and dynamic range clipping.
✓Supports grouped GEMMs to execute all MoE expert calculations in a single unified kernel launch.
✓Under 300 lines of core CUDA kernel code, emphasizing readability and extreme performance.
git clone https://github.com/deepseek-ai/DeepGEMM.git
Repo
Parallelism

DualPipe

Bidirectional Pipeline Parallel Schedule with <10% Bubble Overlap

An innovative bidirectional pipeline parallel scheduling algorithm designed for large-scale distributed training of giant MoE architectures. By running forward and backward micro-batches from both ends of the pipeline simultaneously, it reduces idle bubble time to less than 10%.

Pipeline ParallelismBidirectional ScheduleMoE TrainingSub-10% BubblePyTorch
✓Drastically shrinks the dreaded 1F1B pipeline bubble down to negligible overhead.
✓Interleaves forward and backward passes across dual computation directions without deadlocks.
✓Open-source week described DualPipe as bidirectional pipeline parallelism for V3/R1 training. A 1.7T scaling claim was removed; V4-Pro's card lists 1.6T.
git clone https://github.com/deepseek-ai/DualPipe.git
Repo
Storage & RDMA

3FS & Smallpond

Fire-Flyer File System: RDMA Distributed Storage & DuckDB Data Processor

3FS (Fire-Flyer File System) is a distributed parallel file system designed for AI training clusters, combining raw NVMe-oF throughput with RDMA networking to deliver terabytes/sec of aggregate I/O bandwidth. Paired with Smallpond, a distributed DuckDB-based data processing framework for massive pretraining corpora.

RDMA Distributed FSNVMe-oFDuckDB ProcessingPB-Scale I/OHigh Throughput
✓Streams training datasets across thousands of nodes at 6.6 TB/s peak aggregate throughput.
✓Zero-copy direct RDMA transfer straight from distributed NVMe drives into host RAM.
✓Smallpond processes multi-terabyte data filtering and tokenization using embedded DuckDB engines.
git clone https://github.com/deepseek-ai/3FS.git
Repo
Memory Architecture

Engram

Constant-Time O(1) Hashed Conditional Associative Memory (196B Capacity)

The breakthrough memory architecture integrated into DeepSeek-V4.1-Flash. Engram stores massive factual world knowledge and long-range repository symbols in a 196B parameter hashed associative memory table, retrieved in constant O(1) time without triggering heavy transformer attention layers.

O(1) Associative Memory196B ParametersConditional RecallHash RoutingV4.1 Backbone
✓Decouples factual knowledge storage from transformer attention layers, keeping active parameter count ultra-lean.
✓Constant O(1) retrieval latency regardless of knowledge base size.
✓The V4.1-Flash model card lists Engram conditional memory at 196B parameters. An AIME 96.7 score was removed; it is not in that card's tables.
git clone https://github.com/deepseek-ai/Engram.git
Repo