Open SourceInference84k
vLLM
High-throughput LLM serving engine built around PagedAttention.
vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs, originally from UC Berkeley. Its core innovation, PagedAttention, manages the attention key-value cache like virtual memory to greatly improve GPU memory utilization and batching throughput. It supports continuous batching, tensor and pipeline parallelism, quantization, and an OpenAI-compatible server, and is widely used for production-scale serving.
Repository
vllm-project/vllmLanguage
PythonWhat you'd build with it
- Serving open-weight models at high throughput and low latency in production
- Maximizing GPU utilization for many concurrent inference requests via continuous batching
- Running large models across multiple GPUs with tensor and pipeline parallelism behind an OpenAI-compatible API
Tags
servingthroughputgpu