llama.cpp

Plain C/C++ LLM inference with quantization; runs almost anywhere.

llama.cpp is a C/C++ inference engine for running LLMs efficiently on commodity hardware, including CPUs and a wide range of GPUs. It pioneered the GGUF model format and aggressive quantization (e.g. 4-bit, 5-bit) that lets large models run on consumer machines, and it underpins many higher-level local-LLM tools. It ships a CLI, a server with an OpenAI-compatible API, and bindings used across the ecosystem.