skundu42/recallops

RecallOps is a Python tool built to catch a specific problem: when teams change how they process documents for AI search systems, called RAG pipelines, like switching embedding models, chunking methods, or rerankers, the quality of search results can quietly get worse. Most tools cannot tell you exactly why that happened, only that something changed. RecallOps is built to answer that question with proof, not guesses. The core idea is verified attribution. When search quality drops, RecallOps does not just point at a likely cause. It reruns the pipeline with only one factor reverted at a time and checks whether the failing search recovers. Only if that test passes does it report the cause as verified. Anything it cannot confirm this way is labeled as a hypothesis instead, kept separate from confirmed causes. To make this possible, RecallOps stores full embeddings, not just references to them, so it can replay past searches and test changes without paying to regenerate embeddings each time. It works by versioning an entire document pipeline, including the documents, how they were split into chunks, the embeddings, and the configuration, using content based identifiers so identical inputs always produce the same result. By default RecallOps runs entirely offline at no cost, using local hash based embeddings and a built in exact search method, with no API keys or servers required. Optional extras add support for vector databases such as pgvector, Qdrant, Chroma, and LanceDB, plus embedding providers including OpenAI, Cohere, Voyage, and sentence transformers. Any operation that would cost money through an external provider prints a cost estimate first and requires explicit confirmation. It also includes a continuous integration testing feature that checks retrieval quality against a set of test queries, using statistical methods designed to avoid false alarms from random noise. The project is aimed at teams building or maintaining RAG based search systems who want a way to test changes safely and understand exactly what broke after an update, backed by evidence rather than assumptions. It is licensed under Apache 2.0.