Open-source libraries

Checked on 15 Nov 2026 against deepseek-ai on GitHub.

DeepSeek Open source infrastructure

During DeepSeek's OpenSource Week, the engineering team open-sourced the entire distributed high-performance systems stack powering their frontier clusters. These repositories provide the extreme compute, communication, and memory efficiency required for modern MoE architectures.

github.com/deepseek-ai
Attention & Kernels

FlashMLA

Hopper TMA-Accelerated Multi-Head Latent Attention Kernel

An ultra-fast decoding kernel tailored specifically for Multi-Head Latent Attention (MLA) on NVIDIA Hopper architectures. Employs Tensor Memory Accelerator (TMA) asynchronous memory copies to maximize memory bandwidth utilization during variable-length generation.

Hopper TMAMLA AttentionKV CompressionCUDA C++High Throughput
·Optimized for variable sequence length decoding on H800/H100/H200 GPUs.
·Takes full advantage of Hopper TMA hardware features for zero-copy global-to-shared memory transfers.
·The repository describes optimized MLA kernels for Hopper GPUs and variable-length MLA decoding.
git clone https://github.com/deepseek-ai/FlashMLA.git
Communication & MoE

DeepEP

Ultra-Low Latency MoE NVLink & InfiniBand All-to-All Communication

The first open-source communication library purpose-built for Mixture-of-Experts (MoE) dispatch and combine operations. Implements custom NVLink kernels within nodes and RDMA InfiniBand across nodes, overlapping MoE communication directly with GPU matrix computation.

MoE All-to-AllNVLink CustomRDMA InfiniBandZero BubbleNCCL Alternative
·Eliminates standard NCCL all-to-all serialization bottlenecks across thousands of GPUs.
·Overlaps expert token dispatch and combine with upstream layer tensor calculations.
·Critical communication foundation for both V3, V4-Pro, and V4.1-Flash clusters.
git clone https://github.com/deepseek-ai/DeepEP.git
Compute & GEMM

DeepGEMM

Clean, High-Efficiency FP8 General Matrix Multiplication with Tile Scaling

A clean, performant FP8 GEMM library built specifically for Hopper Tensor Cores. Supports both standard continuous GEMM and fine-grained grouped MoE GEMMs with fine-grained per-tile scaling, achieving over 95% of theoretical peak FP8 tensor core throughput.

FP8 GEMMHopper Tensor CoresTile ScalingCUTLASSQuantization
·Implements fine-grained 128x128 tile scaling to prevent FP8 underflow and dynamic range clipping.
·Supports grouped GEMMs to execute all MoE expert calculations in a single unified kernel launch.
·Under 300 lines of core CUDA kernel code, emphasizing readability and extreme performance.
git clone https://github.com/deepseek-ai/DeepGEMM.git
Parallelism

DualPipe

Bidirectional Pipeline Parallel Schedule with <10% Bubble Overlap

An innovative bidirectional pipeline parallel scheduling algorithm designed for large-scale distributed training of giant MoE architectures. By running forward and backward micro-batches from both ends of the pipeline simultaneously, it reduces idle bubble time to less than 10%.

Pipeline ParallelismBidirectional ScheduleMoE TrainingSub-10% BubblePyTorch
·Drastically shrinks the dreaded 1F1B pipeline bubble down to negligible overhead.
·Interleaves forward and backward passes across dual computation directions without deadlocks.
·Open-source week described DualPipe as bidirectional pipeline parallelism for V3/R1 training. V4-Pro's model card lists 1.6T total parameters.
git clone https://github.com/deepseek-ai/DualPipe.git
Storage & RDMA

3FS & Smallpond

Fire-Flyer File System: RDMA Distributed Storage & DuckDB Data Processor

3FS (Fire-Flyer File System) is a distributed parallel file system designed for AI training clusters, combining raw NVMe-oF throughput with RDMA networking to deliver terabytes/sec of aggregate I/O bandwidth. Paired with Smallpond, a distributed DuckDB-based data processing framework for massive pretraining corpora.

RDMA Distributed FSNVMe-oFDuckDB ProcessingPB-Scale I/OHigh Throughput
·Streams training datasets across thousands of nodes at 6.6 TB/s peak aggregate throughput.
·Zero-copy direct RDMA transfer straight from distributed NVMe drives into host RAM.
·Smallpond processes multi-terabyte data filtering and tokenization using embedded DuckDB engines.
git clone https://github.com/deepseek-ai/3FS.git
Memory Architecture

Engram

Constant-Time O(1) Hashed Conditional Associative Memory (196B Capacity)

The breakthrough memory architecture integrated into DeepSeek-V4.1-Flash. Engram stores massive factual world knowledge and long-range repository symbols in a 196B parameter hashed associative memory table, retrieved in constant O(1) time without triggering heavy transformer attention layers.

O(1) Associative Memory196B ParametersConditional RecallHash RoutingV4.1 Backbone
·Decouples factual knowledge storage from transformer attention layers, keeping active parameter count ultra-lean.
·Constant O(1) retrieval latency regardless of knowledge base size.
·The V4.1-Flash model card lists Engram conditional memory at 196B parameters.
git clone https://github.com/deepseek-ai/Engram.git