miaai-lab/glm-5.2-nvfp4-aqlm-triple-dgx-sparks

This repository provides a set of shell scripts and configuration files for running a very large language model, GLM-5.2, across three NVIDIA DGX Spark machines working together as one unit. The project is built by Mia's AI Lab and targets a specific compressed version of the model that uses two techniques, called NVFP4 and AQLM, to shrink its size from a much larger original down to about 272 GB on disk. This compression is what makes it possible to fit and run the model across three smaller linked machines instead of a single massive server. The setup uses a customized fork of an existing serving program called vLLM. The standard version of vLLM does not understand this particular hybrid compressed model, so the fork adds the specific code paths needed to load and run it correctly. The three DGX Spark nodes communicate over a fast network link called RoCE, and the scripts handle distributing the model weights, starting the coordination layer called Ray, and launching the serving process. The result is a single API endpoint on port 8888 that can accept text requests. The README reports measured performance from a live three node setup. With the default configuration, the system supports a context window of roughly 380,000 tokens, which refers to how much text the model can consider in a single conversation or request. The key value cache, a memory area the model uses to track context during generation, is pinned at 12 GiB per node. This allows a pool of about 386,688 tokens. Reported decode speed ranges from around 13 to 22 tokens per second depending on the content. Memory usage is very tight, with each node sitting close to its 121 GiB limit and sometimes spilling into swap. The README strongly recommends disabling a system utility called earlyoom on all three machines before starting. Because memory is so constrained, earlyoom can mistakenly kill background processes during a critical setup phase called CUDA graph capture, causing failures that look like graphics memory errors but are actually caused by the host system. The project provides shell commands to stop and verify this utility. Requirements include three DGX Spark machines with Docker installed, fast networking between them, SSH access from a head node to two workers, at least 280 GB of free storage per node, and a specific version of a networking library called NCCL. The quick start process involves cloning the repository, editing a configuration file with network addresses and paths, pulling a prebuilt 39 GB container image, downloading the model weights, and running a few scripts in sequence. A start script automates most of the work, checking for missing pieces and launching the Ray coordination layer and the vLLM serving process. The full README is longer than what was shown.