3FS & Smallpond
3FS (Fire-Flyer File System) is a distributed parallel file system designed for AI training clusters, combining raw NVMe-oF throughput with RDMA networking to deliver terabytes/sec of aggregate I/O bandwidth. Paired with Smallpond, a distributed DuckDB-based data processing framework for massive pretraining corpora.
https://github.com/deepseek-ai/3FS
- Streams training datasets across thousands of nodes at 6.6 TB/s peak aggregate throughput.
- Zero-copy direct RDMA transfer straight from distributed NVMe drives into host RAM.
- Smallpond processes multi-terabyte data filtering and tokenization using embedded DuckDB engines.
Plain-language overview
3FS (Fire-Flyer File System) is DeepSeek's distributed filesystem for AI clusters, built around NVMe-oF and RDMA so many training nodes read datasets at aggregate bandwidth measured in terabytes per second. The 3FS repository documents daemons, client libraries, and deployment patterns; Smallpond (in the same ecosystem) uses embedded DuckDB to filter and tokenize massive corpora before training.
Pretraining I/O often bottlenecks before GPU FLOPs do. 3FS targets that layer: striping objects across NVMe enclosures, exposing POSIX-like APIs, and letting Smallpond run SQL-style transforms without copying entire shards to each node. Figures in README/marketing slides should be validated on your hardware — aggregate TB/s requires many RDMA links working in parallel.
API developers may never deploy 3FS, but it explains how DeepSeek fed 14.8T-token runs referenced in the V3 README. Pair with DeepGEMM and DualPipe for the full open training stack index.