electroglyph/ternary_qat

ternary_QAT is a Python tool for shrinking the numbers inside an AI language model so it takes up less space and can run faster, while still letting you keep training it. Normally, the weights inside a language model are stored as regular decimal numbers. This project converts them into just three possible values: negative one, zero, or positive one, grouped in small batches along the model's structure. That conversion follows a format used by a specific family of models called Ternary Bonsai, and the project checks that its output matches that format exactly. It works with models loaded through the Transformers library, and it is specifically built to work alongside a training tool called Unsloth. Once installed, you load a model as usual, then call a function that swaps out the model's linear layers so that the ternary conversion happens automatically every time the model processes data during training. The project supports two main ways of training. You can fine tune the whole model directly, or you can use a technique called LoRA, where the main model is converted to the ternary format and frozen in place while small separate adapter pieces are trained normally in full precision. When you are ready to save a LoRA-trained model, the tool provides steps to properly combine the adapter back into the ternary model. One important detail the author calls out is that training with this ternary approach needs a much higher learning rate than normal fine tuning, roughly ten to fifty times higher. The suggested starting point is around seven ten-thousandths, with room to experiment upward depending on the size of the training data and the settings used. Example code for LoRA training, full fine tuning, and Unsloth usage is included in the project's examples folder. The tool is installed through pip, and users are advised to install their own version of PyTorch first so the wrong version is not installed automatically.