tsuyu122/usaf

USAF, short for Ultra Sparse Adaptive Fine-Tuning, is a Python project that lets someone fine tune large Mixture of Experts, or MoE, AI models on consumer grade graphics cards instead of expensive data center hardware. MoE models are a type of AI model that only activate a small subset of their internal weights, called experts, for any given input, which is what makes it possible to train a fraction of them at a time. The author built this because they only had a consumer AMD graphics card with 12GB of memory, and existing fine tuning methods either would not fit on that hardware or could not train the parts of an MoE model that matter most. USAF trains about 26 million parameters out of a model that has 4.8 billion, while also training the router, the small network that decides which experts handle each piece of input. The project states it is the only method among common options like LoRA, QLoRA, DoRA, and full fine tuning that runs on AMD hardware and touches both the expert weights and the router at the same time. On a test run using the Qwen3-30B-A3B model on a 12GB AMD card, 180 training steps reduced the model's loss by 30 percent and improved performance on both the training data and separate held out data, which the author points to as a sign of genuine learning rather than memorization. The method works by finding the small percentage of weights with the highest impact, periodically swapping out underperforming ones during training using a technique called RigL, and keeping frequently used data in memory to avoid slow repeated calculations. To get started, users install a few Python packages and run a training script, with hardware detected through environment variables for AMD, NVIDIA, or multiple GPUs. The project currently works out of the box with Qwen3-MoE models, and includes a step by step process for preparing a dataset, quantizing expert weights, and configuring a run for other MoE model families such as Mixtral. The author lists larger MoE models they would like to test but do not have the hardware for, and invites people with access to that hardware to reach out. USAF is released under the Apache 2.0 License.