Open SourceInference10k
Triton Inference Server
NVIDIA's production server for any framework, any model, any hardware.
Triton Inference Server is NVIDIA's open-source inference serving platform for deploying models from any framework (TensorRT, PyTorch, TensorFlow, ONNX, Python, and more) on GPUs or CPUs. It supports concurrent model execution, dynamic batching, model ensembles, and exposes HTTP/gRPC endpoints with built-in metrics. It is widely used as the production serving layer for LLMs (via the TensorRT-LLM backend) and traditional ML alike.
Repository
triton-inference-server/serverLanguage
PythonWhat you'd build with it
- Standing up a production inference endpoint that serves models from mixed frameworks behind one API
- Using dynamic batching and concurrent model execution to raise GPU utilization
- Serving LLMs at scale through the TensorRT-LLM or vLLM backends with metrics and autoscaling
Tags
servingnvidiaproductionmulti-framework