mithun50/allm

ALLM is a Rust-based runtime that lets you run large language models on devices without enough RAM to hold the entire model. It targets CPU and RAM instead of requiring a graphics card, which makes it relevant for phones, laptops, and other constrained hardware. The core idea comes from a project called AirLLM, which proved that a transformer model processes its layers one at a time. Each layer is needed briefly, then sits idle until the next word is generated. ALLM streams each layer from storage into a small memory cache, runs the computation, then discards it to make room for the next layer. This means you can run a model that is larger than your available RAM. ALLM adds several things on top of that original concept. It keeps a small cache of recently used layers rather than loading just one at a time. It enforces a strict memory limit so the process never gets killed by the operating system for using too much RAM. A background thread loads the next layer while the current one is computing, which hides storage read delays behind actual work. The project is honest about performance tradeoffs. When a model genuinely exceeds RAM, throughput drops significantly because reading weights from disk every token is slow. Benchmarks show a 0.5 billion parameter model running at about 8 tokens per second when resident, dropping to 3 tokens per second under a tight memory budget. A 3 billion parameter model runs at about 0.5 tokens per second when streamed. The tool is designed for stability and correctness under memory constraints, not speed. The codebase is organized into small, independent modules with clear separation between storage, caching, computation, and the command line interface. The full README is longer than what was shown.