Mooncake
#7 v LLM inference a servingMoonshot AI · od Erste technische Report-Veröffentlichung: 26. Juni 2024; Transfer Engine Open-Source: 28. November 2024; Mooncake Store · 2× · naposledy 19. 7. 2026
Mooncake is an open-source serving infrastructure for large language models, developed by Moonshot AI (operator of the Kimi chatbot) together with the KVCache.AI research group. The architecture separates prefill and decode compute clusters (KVCache-centric disaggregation) and leverages otherwise underutilized CPU, DRAM, and SSD resources of GPU clusters to build a distributed KVCache pool. Core components are the Transfer Engine (RDMA-based data transfer), Mooncake Store (distributed KVCache/weight storage), and P2P Store for checkpoint transfer; the project is Apache-2.0 licensed and listed in the PyTorch ecosystem. In production, Mooncake processes over 100 billion tokens daily across thousands of nodes at Moonshot AI.
Vlastnosti
| Throughput/Latency | 224k tokens/s prefill, 288k tokens/s decode (Kimi K2 on 128 H200 GPUs); Transfer Engine up to 190 GB/s bandwidth (8×400 Gbps RoCE) |
| License | Apache License 2.0 |
| Platform | C++-based framework, Linux servers with GPU clusters (CUDA, MUSA, HIP, MACA, Cambricon MLU, Ascend) |
| Price | Free, open source (no commercial pricing model) |
| Protocol Compatibility | TCP, RDMA (InfiniBand/RoCEv2/eRDMA), AWS EFA, NVMe-oF, NVLink, HIP, Barex, CXL, Ascend transports; integrated with vLLM, SGLang, TensorRT-LLM, LMCache, LMDeploy, NIXL |
| Release Date | Initial technical report: June 26, 2024; Transfer Engine: November 28, 2024; Mooncake Store: March 7, 2025 |
| Supported Models/Providers | Model-agnostic; used in production for Kimi K1.5/K2/K2.5 (Moonshot AI) and in integrations with DeepSeek, LLaMA3-70B |