umarharoon2084/crib-decoding

Crib-Decoding is a research concept for making large language models, the AI systems that generate text, run faster. When such a model generates text one token at a time, it currently scores every word in its vocabulary at each step, even tokens that are clearly impossible given the context so far. Crib-Decoding proposes adding a fast filtering stage before the main model runs, cheaply eliminating implausible tokens so the expensive computation only has to cover a narrowed shortlist. The name comes from Alan Turing's use of cribs during WWII code-breaking. The Bombe machine did not brute-force every possible Enigma configuration, instead it used known phrases and structural constraints to eliminate most possibilities early. Crib-Decoding applies the same logic to language generation. The README also draws parallels to smartphone keyboards like Gboard, SwiftKey, and Apple QuickType, which use tiny models and aggressive pruning to suggest next words on low-power hardware, and to CPU branch prediction in computer processors. The proposed architecture has two stages: a fast crib filter, built from statistical rules, tiny neural networks, or grammar constraints, narrows the token space first, then the full model scores only the remaining shortlist. Potential benefits include lower latency, reduced memory bandwidth, lower energy use, and better performance on edge devices with limited power. Key challenges the author acknowledges include false negatives, where valid tokens get incorrectly removed, creativity suppression in tasks like poetry or fiction, and tokenization complexity, since modern models work with subword fragments rather than whole words. This is an early-stage research concept with no working implementation yet. It is MIT licensed. The full README is longer than what was shown.