woct0rdho/transformers5-qwen3.5-recipe

This repository is a training recipe for fine-tuning large Qwen3.5 and Qwen3.6 language models on consumer-grade graphics cards with limited video memory. It uses a technique called LoRA, which lets you adjust a large model's behavior by training a small set of extra parameters instead of retraining the whole model, saving both time and memory. The key idea is to load the base model in the GGUF format, a compact file format normally used for running models rather than training them. According to the author, GGUF is on track to replace an older format called bitsandbytes for this kind of low-memory training. Using this approach, the author reports training a 35 billion parameter model on 16 gigabytes of video memory with no need to offload work to the computer's regular memory, and even larger models on higher-memory setups. The recipe was built and tuned specifically for a particular AMD GPU architecture called Strix Halo, though the author notes it should be possible to adapt it to other graphics cards with some effort. It combines a long list of technical building blocks: a custom GGUF loader for the Transformers library, compiled dequantization steps to save memory, tuned matrix multiplication kernels, custom Triton kernels for the model's mixture of experts layers and attention mechanism, alternative normalization and loss calculation methods, and a memory-efficient optimizer. This is a highly specialized, code-first project aimed at machine learning practitioners and researchers who are already comfortable with model training internals, GPU kernels, and the Hugging Face Transformers ecosystem. It is not a beginner tool and assumes deep familiarity with the underlying training stack. The README documents the technical choices made rather than offering step-by-step setup instructions, so using this repo requires reading the linked source code and related GitHub discussions to understand how the pieces fit together.