Open SourceInference6k
LMDeploy
Toolkit for compressing and serving LLMs with the TurboMind engine.
LMDeploy is an open-source toolkit from the InternLM team for compressing, quantizing, and serving large language and vision-language models. Its high-performance TurboMind inference engine delivers persistent batching, blocked KV caching, and weight-only/KV quantization for high throughput, and it ships an OpenAI-compatible server. It supports a broad range of model families and is a popular choice for efficient self-hosted LLM serving.
Repository
InternLM/lmdeployLanguage
PythonWhat you'd build with it
- Quantizing LLMs (4-bit weight-only, KV cache quant) and serving them with high throughput
- Self-hosting an OpenAI-compatible endpoint backed by the TurboMind engine
- Serving vision-language models efficiently on a single or multi-GPU setup
Tags
servingquantizationthroughputvlm