leapmemory/memory-bench

memory-bench is an open benchmark tool for testing whether an AI memory service called LeapMemory actually saves on cost and effort compared to the common approach of pasting a full conversation history into an AI model every time. The comparison works by asking the exact same question twice under two conditions: once with the entire past conversation pasted in as context, and once using only a single relevant memory returned by LeapMemory. Both answers are checked against a known correct answer, and only questions that both approaches answer correctly count toward the final comparison, which the project says prevents a memory failure from being hidden in the numbers. The results show real token savings, growing larger as conversations get longer, since pasting the full history grows with every turn while the memory-based approach stays roughly the same size no matter how long the conversation gets. In the tests reported, the memory-based approach used significantly fewer tokens than pasting full history while matching accuracy exactly. The tool tests direct factual recall, such as names, numbers, dates, and one changed fact, not deeper reasoning, contradiction handling across many facts, or performance after months of real use, and the author states these gaps plainly rather than hiding them. Running the tool yourself requires signing up for a LeapMemory account, creating an API key, adding a small amount of prepaid credit, and running a provided Python script against either a public sample conversation or your own exported chat history. The project is written in Python and released under the MIT license, meaning it can be freely used, copied, and modified, including for commercial purposes.