Ray Serve / Ray

Distributed framework for scaling model serving and AI workloads.

Ray is an open-source framework for scaling Python and AI applications, and its Ray Serve library provides a scalable, framework-agnostic model serving layer. Ray Serve supports composing multiple models into deployment graphs, autoscaling, fractional GPU allocation, and batching, and includes LLM-specific serving with an OpenAI-compatible API backed by engines like vLLM. It is widely used to serve and orchestrate inference at scale across clusters.