DeepInfer
#4 en Inférence & serving LLMUnknown · depuis 2026-10-20 · 3× · vu le 23 oct. 2026
46
Momentum
DeepInfer is an open-source inference framework from DeepSeek (deepseek-ai/DeepInfer on GitHub) that runs DeepSeek-V4.1-Flash for offline inference, according to the repository. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs under Linux and is licensed under Apache-2.0. The KV cache system, prefill/decode disaggregation and load balancing are not part of the release.
Historique du momentum
26.07.24.10.
Fonctionnalités
| Throughput/Latency | Decode above 25,000 tokens/s/GPU on 64 GPUs (self-reported, not independently verified) |
| License | Apache-2.0 (code); model weights MIT |
| Platform | Linux; GCC >= 13.0, Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend) |
| Protocol Compatibility | Structured outputs: regex, JSON Schema, EBNF; no public chat API included |
| Release Date | October 20, 2026 |
| Supported Models/Providers | DeepSeek-V4.1-Flash (only supported model) |