tonyd2wild/minimax-m3-2x-dgx-spark-36-tok-s

This repository documents how to run the full, unshrunk version of a very large AI language model called MiniMax-M3, which has 428 billion parameters, on a pair of small NVIDIA computers designed for AI work called DGX Sparks, and have it respond at a usable speed of about 36 words per second. Normally a model this size needs much bigger, more expensive hardware, so the project explains the specific tricks used to make it fit and run reasonably fast on two of these smaller machines connected together. The author is upfront that this work builds directly on someone else's published setup rather than starting from scratch. That original work, licensed under MIT, provided the core recipe, and this repository reproduces it on independent hardware, confirms the original speed claims hold up, fixes two build and deployment problems, recovers a configuration the original author had set aside, and documents tricky gotchas encountered along the way. Technically, the model's memory footprint is shrunk using a 4-bit weight compression technique, its short-term memory cache is also compressed to 4 bits, and a smaller helper model predicts likely upcoming words to speed up generation, a technique called speculative decoding. All of this runs through a serving tool called vLLM, split across two machines working together. The project offers three different serving configurations to choose from depending on whether a user wants more speed, more conversation length capacity, or a balance of both, with the balanced option being the one the author has fully verified. Setup involves cloning the original author's code, applying two provided patches to fix build issues, building a Docker container image, downloading the large model files once and copying them between the two machines over a fast network link, then starting the server with a specific configuration file. The README includes exact commands for each of these steps, along with a warning to discard the very first request after starting the server, since it takes about a minute to fully warm up before giving an accurate sense of speed.