Vllm · od Juni 2023 (offizieller Launch-Blogpost); zugehöriges Paper September 2023 · 59× · naposledy 13. 8. 2026
100
Momentum
vLLM is an open-source inference and serving engine for large language models, originally developed at UC Berkeley's Sky Computing Lab. Its core innovation is PagedAttention, a memory-management technique for the KV cache, combined with continuous batching, enabling high throughput and memory-efficient model serving. The project supports over 200 HuggingFace model architectures, numerous quantization formats, and various hardware backends (NVIDIA, AMD, Intel, TPU, Trainium). vLLM is distributed as free open-source software under the Apache 2.0 license and can be self-hosted or run via an OpenAI-compatible HTTP server.
Vývoj momenta
19.05.17.08.
Vlastnosti
| Deployment (Self-host/Cloud) | Self-hosted via pip/uv-Installation oder Docker-Image (vllm/vllm-openai); K8s-natives Deployment über production-stack |
| Durchsatz/Latenz | Bis zu 24x höherer Durchsatz als HuggingFace Transformers (offizieller Launch-Benchmark 2023) |
| Lizenz | Apache License 2.0 |
| Plattform | Python-Bibliothek/Server; unterstützt NVIDIA-, AMD-GPUs, Intel-CPUs, TPU, AWS Trainium/Inferentia, Apple Silicon (via vLLM-Metal); Python 3.10–3.13 |
| Preis | Kostenlos, Open Source (keine Lizenzgebühren; Kosten entstehen nur durch selbst betriebene Infrastruktur) |
| Protokoll-Kompatibilität | OpenAI-kompatibler HTTP-Server (Chat Completions, Responses API, SageMaker-kompatibler Endpunkt) |
| Release-Datum | Juni 2023 (Launch-Blogpost 'Easy, Fast, and Cheap LLM Serving with PagedAttention') |
| Unterstützte Modelle/Provider | 200+ Modellarchitekturen auf HuggingFace (u.a. Llama, DeepSeek, Qwen); Quantisierung: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF |