opencpil/prima.cpp

prima.cpp lets you run very large AI language models, the kind with 30 to 70 billion parameters, across a group of your own everyday devices instead of needing an expensive server. It builds on the well known llama.cpp project, but spreads the work across multiple machines you already own, such as a laptop, a desktop, and even a phone or tablet, whether or not each one has a dedicated graphics card. The main problem it solves is that models this large usually run out of memory or slow a device to a crawl. prima.cpp keeps memory use under about 10 percent by loading model data from disk only as needed, so you can run a huge model while still using your device for other things. The README reports that combining several modest home devices this way can make large models roughly 15 times faster than running them the normal way on a single machine, with example numbers like Llama 3 70B producing over a token and a half per second, similar to a slow audiobook narrator. Several technical ideas make this work. It uses memory mapping so the operating system only loads pieces of the model as they are needed, and it can split work between a device's graphics card and its regular processor when both are available. It also uses a specialized pipeline technique to overlap loading data from disk with actual computation, and a scheduler that looks at each device's processing power, disk speed, memory, and operating system to decide how much work to give it. Weak devices can be automatically detected and skipped if removing them would speed things up. The project supports several ways of shrinking models to save space, and it can also use techniques like speculative decoding and batching to further speed up responses for multiple users at once. It currently supports popular open model families including Llama, Qwen, and DeepSeek, and can run on macOS, Linux, Android through Termux, and HarmonyOS through Termux, with Windows support planned. The project is released under the MIT license, meaning it can be freely used, modified, and redistributed as long as the copyright notice is kept. Anyone who wants to run large AI models at home without buying expensive server hardware is the intended audience here, though comfort with command line tools and basic networking is assumed.