poisonxa16/pxq_llama

pxq_llama is a fork of ik_llama.cpp, itself a variant of the popular llama.cpp project for running large language models locally. This project focuses specifically on making older, cheaper NVIDIA GPUs, the kind sold secondhand as surplus rather than current top of the line cards, useful for running large modern AI models that would normally need much more expensive hardware. The project does this in two parts. First, it fixes and speeds up the underlying engine so that any mixture of experts model, a type of large model that only activates part of itself for each request, runs faster on older Pascal and Volta generation GPUs, whether on a single card, spread across several cards, or partly offloaded to a computer's regular memory. Second, it introduces its own family of compression formats called PXQ, which shrink a model's expert layers down to two, three, or four bits per value instead of the usual higher precision, paired with custom CUDA code tuned for that older hardware. Together these let a 35 billion parameter model run on a single graphics card with only twelve to sixteen gigabytes of memory, something around 150 dollars can buy secondhand, and let much larger models in the 120 billion parameter range run across a stack of older cards. The README includes detailed, reproducible benchmark comparisons against the original ik_llama.cpp project, measuring both how fast the model processes an initial prompt and how fast it generates new text afterward, on specific card models like the P100, V100, and 1080 Ti. Recent updates describe renaming its quantization tiers for clarity, fixing a bug in an experimental kernel, and adding new optional environment variables that trade a small amount of numerical exactness for a large speed increase on certain older cards. Building the project requires a CUDA toolchain and is aimed squarely at people already comfortable running large language models locally and tuning GPU inference performance, rather than newcomers.