swellweb/reame

Reame is an LLM inference server built on top of llama.cpp and designed specifically for cheap CPU hardware, such as a free tier cloud box, a shared virtual server, or a small 2-core ARM machine, rather than treating CPU as a fallback when a GPU is unavailable. Its guiding idea is that on a CPU, the server should never compute the same thing twice. It is aimed at narrow, repetitive workloads run over your own data on hardware you already have, such as document extraction, batch tagging jobs, private code autocomplete, or adding AI features to a small application without paying for a metered API. It is explicitly not meant as a general chatbot replacement, an agentic coding assistant, or a tool for long creative writing, since those tasks benefit from much larger models than the ones Reame targets. The server's main trick is a persistent cache: prompt prefixes are saved to disk in compressed, checksummed form and reused across different prompts, across restarts, and across separate server processes, so a shared system prompt is only computed once. A related feature called Palimpsest keeps an archive of everything the server has generated before, and can draft new answers from that archive at no extra cost. Another feature speculatively guesses upcoming tokens using either a small draft model or free pattern matching from the prompt itself, and automatically turns this off if measurements show it is not actually helping on the current hardware. An optional consensus mode can generate several candidate answers to the same prompt in one batch and pick the majority answer, which the project describes honestly as reducing random mistakes rather than making a small model reason as well as a larger one. Reame exposes an OpenAI compatible REST API with streaming, sessions, and authentication, so any existing OpenAI client library can point at it, and it includes a simple command line tool that downloads a model and starts chatting with one command. It also offers a separate shared memory service called ARCA that multiple Reame instances can connect to for a shared response cache. The project includes a large automated test suite and publishes its own benchmark numbers, including cases where its approach did not help, on the project's GitHub Pages site.