sapid-labs/vllm-moet

This project is a patched version of vLLM, a popular tool for running large language models, aimed at making a very large AI model called DeepSeek-V4-Flash work on NVIDIA's newer Blackwell graphics cards. According to the README, the official vLLM release is broken for this model on this hardware, so the maintainers wrote a large patch, over 5,600 lines, to fix it and also to shrink the model so it fits on cards that would otherwise not have enough memory. DeepSeek-V4-Flash is a 159 billion parameter model, and the trick used here is compressing most of its internal building blocks down to 2 bits instead of the usual higher precision, while keeping accuracy close to a 4-bit version through a recovery technique. The README explains that a naive version of this compression breaks the model, producing repetitive nonsense, and traces the cause to an imbalance in how positive and negative values were represented. Fixing that imbalance restores quality while keeping the smaller size. The README reports specific performance numbers on different graphics card setups, including running the full 159 billion parameter model on a single high-end consumer card by treating the GPU's memory as a cache for the most frequently used parts of the model, while the bulk of the compressed model sits in regular system memory. It also describes applying the same technique to an even larger model, GLM-5.2, across two cards. The repository includes hand-written low-level GPU kernels, a Docker build for quick setup, and a runbook document referenced for a specific hardware variant, alongside detailed documentation of the underlying fixes to the base vLLM release. This is deeply technical, research-grade software intended for people already comfortable running large AI models on specialized hardware, rather than a beginner tool.