vLLM

#1 v LLM inference a serving

vllm · od Juni 2023 (offizielles erstes Release) · 40× · naposledy 30. 6. 2026

100

Momentum

vLLM is an open-source inference and serving engine for Large Language Models (LLMs), originally developed at UC Berkeley's Sky Computing Lab and maintained as a community project since 2023. Its core architecture is based on PagedAttention (virtual memory management of the KV cache) and continuous batching, delivering significantly higher throughput than naive serving approaches. vLLM supports 200+ model architectures from Hugging Face and runs on a broad range of hardware accelerators. The project is free to use (Apache 2.0) and is backed by an ecosystem of over 2,000 contributors and sponsors including NVIDIA, AMD, Google, AWS, and Intel.

Zdroje (40)