kacper-daftcode/vllm-moet

This project is a large patch for vLLM, a popular tool for running AI language models, that makes it possible to run some of the biggest and most capable open AI models on consumer or workstation graphics cards instead of expensive data center hardware. The models it targets, GLM 5.2, DeepSeek V4 Flash, and Kimi K2.7 Code, are normally far too large to fit in the memory of a single gaming or workstation GPU. The way it manages this is by shrinking the size of certain parts of these models, called experts, down to a much smaller two bit format instead of the usual higher precision the model was trained with. Shrinking things this aggressively normally breaks a model's output quality, so the project also includes a system that keeps a smaller set of these experts at a higher precision level and swaps them in when needed, recovering most of the lost quality. When even that is not enough to fit in the graphics card's memory, the project can also store parts of the model in regular computer memory or on a fast storage drive, treating the graphics card like a cache that fetches pieces of the model only as they are needed. On top of this compression system, the project fixes bugs in a specific version of vLLM so it actually works correctly on newer Nvidia graphics card chips, and it adds support for faster response generation and a more efficient way of storing conversation history in memory. The README includes detailed performance numbers for running these massive models across different combinations of graphics cards, showing how many words per second each setup can generate and how much context, meaning how much text, the model can consider at once. This is a specialized project for people who already work with running large AI models and have access to high end but still consumer grade graphics hardware. It includes hand written low level code for the specific graphics chips it supports, so it is not a general purpose tool, it is built for a narrow but technically demanding use case.