KTransformers

Run huge MoE models locally by offloading experts to CPU and GPU.

KTransformers is an open-source framework from Tsinghua's KVCache.AI for optimized local LLM inference, focused on running very large mixture-of-experts models on modest hardware. It uses heterogeneous CPU/GPU offloading and kernel optimizations to serve models like DeepSeek-V3/R1 with only one or a few consumer GPUs plus system RAM, exposing an OpenAI-compatible API and a chat UI. It became widely known for making frontier-size MoE models runnable on a single workstation.