quixiai/bitnet

This repository is bitnet.cpp, an inference framework built for running so called 1-bit large language models, most notably a model family called BitNet b1.58. Regular AI models store their internal numbers using many bits of precision, which takes up a lot of memory and computing power. BitNet models instead compress those numbers down to roughly 1.58 bits each, and this project provides the specialized code needed to run such compressed models quickly on ordinary computer hardware, both CPUs and, more recently, GPUs, with NPU support planned next. The project reports large speed and efficiency gains from using these specialized kernels compared to a naive implementation. On ARM based CPUs it reports speedups between about 1.4 and 5 times, along with meaningfully lower energy use, and on x86 CPUs the reported speedups are even larger, alongside similar energy savings. The README also claims that a 100 billion parameter BitNet model can run on a single CPU at a pace comparable to a person reading text aloud, which is presented as a meaningful step toward running very large language models on ordinary local devices rather than in the cloud. The project builds on top of the well known llama.cpp inference engine and reuses lookup table techniques originally developed in a separate Microsoft project called T-MAC. It ships support for a small number of officially released BitNet models directly from the team behind the technique, alongside several existing 1-bit models hosted on Hugging Face by outside groups, with a table in the README indicating exactly which kernel types are supported on x86 versus ARM chips for each one. To build it yourself, you need Python 3.9 or newer, the CMake build tool, a recent version of the Clang compiler, and ideally the conda package manager for handling dependencies. On Windows, this means installing Visual Studio with several specific components enabled, including C++ development tools and the Clang compiler toolset. Once those requirements are in place, building from source starts with cloning the repository, including all of its nested submodules.