DeepInfer
#3 in LLM Inference & ServingDeepSeek · since 2026-10-20 · 4× · last seen Oct 21, 2026
68
Momentum
DeepInfer is an in-house inference framework open-sourced by DeepSeek, which the company says powers its production services. The release covers offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs under the Apache 2.0 license. A complete serving stack (KV cache system, prefill/decode separation, load balancing) is not included.
Momentum trend
24.07.22.10.
Features
| Throughput/Latency | Self-reported by vendor: decode above 25,000 tokens/s/GPU on 64 GPUs (not independently verified) |
| License | Apache 2.0 (code); model weights MIT per MarkTechPost |
| Platform | Linux, GCC 13+, Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Huawei) |
| Protocol Compatibility | Companion deepseek-recipe converts requests into token sequences using the same logic as the official DeepSeek API |
| Release Date | October 20, 2026 |
| Supported Models/Providers | Only DeepSeek-V4.1-Flash (default checkpoint DeepSeek-V4.1-Flash-1017) |