sahel-sh/deephone

DeepHone is the research code behind an academic paper called Rerank Before You Reason, accepted at ACL 2026 Findings. It studies AI agents that search through documents and reason about them to answer hard questions, sometimes called deep search agents. The question the paper asks is where an agent's limited compute budget is best spent: on having the AI think longer at search time, or on reranking, a step that reorders retrieved documents by relevance before the agent starts reasoning about them. To test this, the authors use a benchmark called BrowseComp-Plus, made up of 830 questions that require real reasoning, searched against a fixed set of about one hundred thousand documents. They separately measure the retriever that finds candidate documents, the reranker that reorders them, and the agent that reasons over the results, and compare the tradeoffs using a new measurement they call Effective Token Cost. Their main finding is that reranking reliably improves both how relevant the retrieved documents are and how often the agent gets the final answer right, and that a moderate amount of reranking is often a cheaper way to gain accuracy than making the agent reason more. They also find that small, efficient rerankers capture most of these gains at a much lower cost than larger ones. This repository builds on an existing project called BrowseComp-Plus, keeping its benchmark, its retrieval methods, and its agent clients for services including OpenAI, Anthropic, Gemini, and others, while adding the new reranking integration and cost analysis used in the paper. It provides two kinds of rerankers, one that reranks whole lists of documents at once and one that judges documents one at a time, several retrieval methods including keyword based and embedding based search, scripts that use another AI model as a judge to score answers, and scripts that generate the charts and tables from the paper. Running the full pipeline requires a fairly involved setup: a Python environment managed with a tool called uv, a Java runtime for the keyword search component, and access to GPUs to run the large language models locally using a serving tool called vLLM. The README documents the exact commands needed to reproduce each table and figure from the paper.