apoorvumang/retrieval-vs-recall

Retrieval vs recall is a research repository that asks a pointed question about AI benchmarks: what happens to a search benchmark's score if you simply turn the search off? Many so called agentic search benchmarks test whether a model can look up information on the web and answer a question with it, but if the facts being asked about are old enough to have been part of the model's original training data, the model can just answer from memory without actually searching anything, and still score well. That would make the benchmark measure memorization rather than real search skill. The repository backs this claim with a specific comparison on a benchmark called WideSearch. One model scored the same, or even slightly better, when search was disabled entirely compared to when it used the full search pipeline. It also walks through a single clear example task, asking for UK house price statistics that must be cited from government websites, where the model with search access returned a blank answer while the same model answering from memory alone produced a plausible looking answer that the grading system rewarded, even though it was not actually retrieved from any source. The same pattern shows up on a second benchmark, SealQA, where one model's apparent lead disappears once the questions are restricted to events after its training cutoff. The code lets you reproduce these findings from data already committed in the repository, without needing any API keys, using a single script. For people who want to test their own model, there is also a test tool that runs the same staged search pipeline, planning, web search through the Exa API, and answer synthesis, with a flag to disable search entirely and compare the two conditions using the benchmark's official grading tool. This project is mainly useful for AI researchers and engineers evaluating how much a benchmark's scores reflect genuine retrieval ability versus a model simply recalling facts it already knows.