reezanahamed/meaningcache

meaningcache is a semantic cache for calls to a large language model that runs with no torch, no separate server, no API key, and no GPU. It is a single readable Python file that uses small static embeddings to compare the meaning of a new prompt against previously stored prompts, so a rephrased question like can I get a refund can be matched to a stored answer for what is the policy on refunds instead of triggering another expensive model call. The README is upfront about the tradeoff involved. Tested on 5,000 real question pairs from a public dataset, the tool serves about 70 percent of reworded questions from cache at its default similarity threshold, but it also wrongly matches about 28 percent of genuinely unrelated questions at that same setting. The threshold is the one adjustable setting, and raising it catches fewer rephrasings while lowering the rate of wrong matches, so the project frames itself as a cost optimizer rather than a correctness guarantee. Using it means creating a SemanticCache object with a chosen threshold, checking it for a match before calling your model, and storing the result afterward, or wrapping a function so this happens automatically. Internally, each prompt is turned into a vector using a small CPU only model called model2vec, about 60 megabytes in size, and a lookup compares that vector to all stored prompts by similarity, which the README says takes about 0.1 milliseconds and can process tens of thousands of prompts per second on a laptop CPU. The README lists clear limitations: it can return a wrong cached answer when two prompts are close in meaning but actually need different answers, it works best on prompts whose correct answer does not depend on time or personal context, and its lookup speed is linear in the number of stored prompts, so very large caches would need a proper vector index instead. It has only been tuned and tested on English prompts. Installation is a simple git clone followed by running a demo script with the uv tool. The project is released under the MIT license.