bigattichouse/packed-twin-inference

Packed Twin Inference is a technique and codebase for making a locally-run AI language model generate text significantly faster, without changing the quality or accuracy of what it produces. The target audience is people who run large AI models on their own hardware and want to get more tokens per second out of the same GPU. The core problem it addresses is that modern large language models are slow because each word they generate requires loading roughly 25 gigabytes of model weights from GPU memory, and the GPU's actual computing units sit mostly idle during that load. The insight is that verifying a batch of 16 or 32 guesses in a single pass costs only a few times more than generating one token, not 16 or 32 times more. So if you can guess upcoming tokens cheaply, you batch-verify those guesses and accept all the correct ones at once, dramatically increasing throughput. The project combines two sources of free guesses. First, a text lookup: when the model is re-typing something already in the conversation (such as editing code it just wrote, or quoting a document), the continuation can often be found by simple string matching. Second, the specific AI model this is tested with (Qwen3.6-27B) has a small built-in predictor that guesses the next token with about 85% accuracy at a fraction of the cost of a full pass. Both sources are combined so that when they disagree, the built-in predictor wins. A safety mechanism verifies every guess against the model's actual output before emitting it, which means wrong guesses are silently discarded and the final text is byte-for-byte identical to what the model would have produced without any of this acceleration. The project includes a test that deliberately corrupts drafts and confirms the output still matches the baseline. In practice, the speedup depends heavily on what is being generated. Editing and refactoring existing code benefits the most, reaching about 2x faster in measurements. Generating fresh prose with no overlap to the prompt gives a more modest improvement around 1.2x to 1.4x. In the worst case, when every guess is wrong, the system shuts speculation off and runs at roughly normal speed. The code is built on top of llama.cpp, the popular open-source library for running AI models locally. It exposes an OpenAI-compatible server API and a command-line chat interface.