sharma-open-source/nunspark

NunSpark lets you run large AI language models on a Mac even when the model is too big to fit in your computer's memory. Normally, if a model needs more memory than your Mac has available, you simply cannot run it. NunSpark works around this by splitting the model into smaller pieces stored on your hard drive, then loading only the specific pieces needed for each word it generates, rather than trying to hold the whole thing in memory at once. Some AI models are built so that only a small portion of the model actually does the work for any given word, a design called mixture of experts. NunSpark takes advantage of this by only loading the specific active portions from disk instead of the entire model, which makes it much faster than simply streaming everything in order. It also uses a technique where a smaller, quicker draft model guesses ahead several words at a time, and the larger model checks those guesses in one pass instead of working word by word, which saves a lot of time reading data off the disk. Despite these shortcuts, the maker states that the output is exactly the same as what you would get running the full model normally, aside from occasional and self-correcting rounding differences. The project includes real performance measurements on Apple Mac computers with Apple Silicon chips, showing that models needing far more memory than a given Mac has can still generate readable text at a usable pace, though far slower than a machine with enough memory to hold the whole model at once. Which settings work best depends heavily on how much memory your specific Mac has. To try it, you need a Mac with Apple Silicon, Python version 3.11 or newer, and a fair amount of free disk space, since both the downloaded model and a repackaged copy of it are stored temporarily. Installation and first use only take three commands. The author is upfront that this is an experimental research project rather than a finished, production-ready tool, and provides detailed write-ups of what worked and what did not during development. It is released under the MIT License, so it can be freely used, copied, and modified.