thomasblc/qwen-ondevice-bench

This project is a small benchmark that tests three versions of the Qwen AI model on hard math reasoning problems, run entirely on one Apple laptop with nothing sent over the internet. The author's main point is not really which model wins, but a set of testing mistakes they found along the way that can quietly ruin any small local model evaluation if you are not careful. The biggest finding is that a setting called decoding temperature, which controls how random a model's answers are, matters enormously. With the usual random settings, the smaller models looked like they could barely solve anything. Switching to a fully deterministic, no randomness setting flipped many of those failing problems into solved ones. In other words, a lot of what looked like the model being incapable was actually just noise from randomness, not a true measure of ability. The benchmark also found that a medium sized 4 billion parameter model and a larger 9 billion parameter model performed almost identically on these reasoning problems, with the bigger model never clearly winning. Only a third model, a 35 billion parameter model that uses a technique where just a small portion of it activates for each word generated, was able to solve the hardest problems involving long, exact step by step calculations. Interestingly, that model activates fewer parameters per word than the 9 billion parameter model does, so its advantage seems to come from its overall training and size rather than doing more work per step. The repository includes the exact problems used, the code that calculates correct answers ahead of time using plain Python, the full text each model produced, and simple scripts to reproduce every result yourself. The author is upfront that this is a small, hand picked test meant to illustrate a point about evaluation method, not a rigorous scientific benchmark, and points readers to larger established math benchmarks for that purpose. The project is released under the MIT License.