eriktromp/colibri-hy3

This project lets you run a very large AI language model, Tencent's Hy3, on an ordinary consumer computer instead of the expensive server hardware such a model would normally need. Hy3 has 295 billion parameters in total, but it is built as a mixture of experts model, meaning only a smaller portion, about 21 billion parameters, is actively used for any single response. This tool takes advantage of that by keeping only a small always-needed part of the model, around 5 gigabytes, loaded in memory, while the rest of the model's specialized parts stream from your hard drive or solid state drive as needed, and optionally from your graphics card's memory too. The project is a port of an existing engine called colibri, rewritten to support this specific model, and it is written entirely in C with no external dependencies. It includes the inference engine itself, a tool for converting the model's original weight files into a compressed format this engine can read, and a command line tool called coli for chatting with the model, running it as a server, converting weights, planning memory usage, and diagnosing setup problems. The README explains that running it requires downloading a large, pre-converted version of the model from Hugging Face, roughly 142 gigabytes on disk, since the engine does not work with common formats like GGUF. It walks through different memory setups: a lean profile for a 16 gigabyte RAM machine with a 16 gigabyte graphics card, a more comfortable profile for machines with 32 to 62 gigabytes of RAM, and a CPU only option with no graphics card at all, which is slower but still works. There is also a long list of tuning options for advanced users, covering things like context length, disk read methods, and memory caps. This project would suit developers or hobbyists who want to experiment with a very large language model on their own hardware without needing a data center, and who are comfortable working with the command line and compiling C code.