swival/ds4-m5

DwarfStar 4 is a small, purpose built inference engine for running the DeepSeek V4 Flash AI model locally on your own machine, written in C. Unlike general purpose AI runners that aim to support many models, this project takes a deliberately narrow approach: it runs one model, DeepSeek V4 Flash, and aims to run it well on high end personal machines. The engine targets Metal on macOS and CUDA on Linux for hardware accelerated processing. One of its defining features is treating the KV cache, the memory store that holds context as the model generates text, as a disk resident structure rather than purely a RAM object. DeepSeek V4 Flash has a highly compressed KV cache, and combined with fast SSD storage in modern Macs, this allows the engine to handle very long context, up to 1 million tokens, and even persist that context across sessions. This makes long context inference feasible on machines with 96 or 128 GB of RAM using 2-bit quantization. The project ships with its own specially crafted model weight files in GGUF format, a download script to fetch them from Hugging Face, and an HTTP server API so the model can serve requests like a local API endpoint. The quantization strategy keeps quality reasonably high by applying aggressive 2-bit compression only to the routed mixture of experts layers, leaving other components at higher precision. The code is alpha quality, the authors note it was developed with strong AI assistance and has only existed for a short time. It is not a general GGUF loader and only works with the specific weight files published for this project, so it will not load arbitrary DeepSeek or GGUF files. The full README is longer than what was shown.