sxuff/nemotron-puzzle-mlx

This project lets you run a large language model called Nemotron Puzzle 75B on an Apple Silicon Mac with 64 GB of memory, using a technique called mixed precision quantization to shrink the model enough to fit. Quantization means storing the model's numbers with fewer bits than normal, which makes the file smaller and faster to run, at some cost to accuracy. The model is a mixture of experts design, meaning most of its size comes from a large bank of specialized sub networks that are only partly used for any given word it generates. The author found that applying a more aggressive four bit compression specifically to that expert bank, while keeping the rest of the model at a higher precision, gave the best balance. Tested on an Apple M2 Max, this four bit expert version produced a smaller file, faster generation speed, lower memory use, and better results on a set of test tasks compared to a five bit version. To use it, you need support for this specific model type added to the underlying MLX library, which the author has proposed as a pull request and made available as a separate branch you can install in the meantime. You can either download the already converted model checkpoint from Hugging Face, or convert your own copy of the original model yourself using the included scripts. The repository also includes scripts to verify that generation is working correctly and to run a set of benchmark checks comparing different quantization settings. A notable part of this project is a bug fix the author found and documented, where a specific calculation inside the model needed to be done in a higher precision format to match the original results correctly, which they explain in detail in an included engineering report. The repository itself contains only code and benchmark results, not the actual model weights, which must be obtained separately and are subject to their own license. The code in this repository is released under the MIT License.