hiceron/spark-nvfp4-lab

This project is a set of real world speed test results for running large AI language models on a specific piece of hardware called the DGX Spark, a compact machine built around Nvidia's GB10 chip. The author spent four days trying to reproduce a claimed 1.79 times speed boost from a special low precision number format called NVFP4, and this repository is the write up of what actually happened when the settings were tested one at a time on one physical machine. The main finding is that a particular fast processing mode, meant to speed up how the model handles its internal expert routing, works fine when combined with a token prediction shortcut technique, once a specific hidden configuration option is set correctly. However, once running, that fast mode actually made responses about 11 percent slower on this hardware and cut the amount of conversation history the model could hold in memory to about a third of what it was before. The author explains this happens because on this chip, the slow part of generating each response is limited by how fast data can move in and out of memory, not by the expert routing step the fast mode was designed to speed up. A second finding is that the time it takes to get the very first word of a response back can grow drastically as the input prompt gets longer, something the author says most published benchmarks never show. A third finding compares a smaller specialized model against a larger general purpose model, showing the smaller one runs three to four times faster for only a tiny difference in a standard reasoning test score. The repository includes the exact deployment commands used, a full table of every measurement taken, a list of five real crashes encountered while testing, and notes on how the tests were run so others can repeat them. The scripts in the repository are released under the MIT License, while the author asks that anyone reusing the measurements themselves credit the source.