efschu/htsglang

htsglang is a modified version of an existing project called sglang, which is a serving system that runs large AI language models so they can answer requests quickly. This fork focuses on one specific hard problem: making a language model run efficiently when it is split across graphics cards, called GPUs, that do not all have the same amount of memory, or when more than one slice of the model shares a single physical GPU. Normally, when a large model is split across several GPUs to share the work, each GPU is given an equal sized piece of the model. That works fine if every GPU is identical, but it wastes the extra memory on a bigger card if the cards do not match. This project's main feature lets the model be split unevenly instead, giving a bigger slice of the model to a GPU with more memory and a smaller slice to a GPU with less, calculated automatically based on how much free memory each card actually reports. This uneven splitting is applied carefully across every part of the model that needs to be divided, including its attention mechanism and its memory cache for ongoing conversations. A related feature lets a user explicitly choose which physical GPU each piece of the split model runs on, including letting two pieces share the same card if there is room. The project also adds support for loading a particular family of models, called Qwen3.5 and Qwen3.6, from a compressed file format called GGUF, something the original sglang project cannot do for these particular models. Loading these compressed files correctly requires undoing several changes a conversion tool makes to the internal structure of the model, which this fork handles automatically. It also supports a speed up technique called speculative decoding using a smaller draft model bundled inside the same GGUF file, and it supports a tiered memory cache system that can spill older cached data out to system memory and disk instead of losing it. The project ships two ready made Docker container images so a user does not need to build anything from source, and it includes example commands for launching a model server with these features enabled. The project is written in Python and builds on top of the original sglang project.