SGLang

Fast serving framework for LLMs and vision-language models.

SGLang is a fast serving framework for large language and vision-language models that combines an optimized runtime with a structured generation language. Its runtime features RadixAttention for automatic reuse of shared prefix KV caches, along with continuous batching and efficient structured/constrained decoding. It targets high-throughput serving of complex, multi-call and agentic LLM programs.