Open SourceInference11k
TensorRT-LLM
NVIDIA's library for optimizing and serving LLMs on its GPUs.
TensorRT-LLM is NVIDIA's open-source library for compiling and running optimized LLM inference on NVIDIA GPUs. It provides a Python API to define and build highly tuned TensorRT engines with features like in-flight batching, paged KV caching, FP8/INT4 quantization, and tensor/pipeline parallelism. It pairs with the Triton Inference Server backend to deliver some of the lowest-latency LLM serving on NVIDIA hardware.
Repository
NVIDIA/TensorRT-LLMLanguage
C++What you'd build with it
- Compiling LLMs into optimized TensorRT engines for the lowest latency on NVIDIA datacenter GPUs
- Serving quantized (FP8/INT4) models with in-flight batching to maximize GPU throughput
- Deploying multi-GPU LLM inference with tensor and pipeline parallelism behind Triton
Tags
servingnvidiagpuquantization