albertofettucini/faithgate

FaithGate is a testing tool for apps built around large language models, like chatbots that answer questions using retrieved documents. It checks whether each answer is actually supported by the information it was given, then compares that score against an earlier version of the same test suite. If a change to a prompt or a model causes answers to become less accurate, the tool fails the build in continuous integration, similar to how a regular test suite fails when code breaks. The tool runs entirely on your own machine and does not send data anywhere except to the judge you choose to score answers. By default that judge is Claude, using an API key you provide yourself, but a fully local scoring mode is also available using an on device model, and the project is upfront that this local mode is weaker than the Claude based one. There is also a basic offline scorer meant only as a placeholder, since the authors do not trust it to give meaningful results on its own. A test suite is a simple text file where each line describes one question, the answer given, and the passages that were retrieved to support that answer. FaithGate scores each line, then compares scores between two labeled runs, showing which questions regressed, improved, or stayed the same. It refuses to pass a build if too few cases could be scored or if every case abstained from scoring, so a broken suite cannot accidentally show green. The project describes itself as a narrow companion to broader evaluation tools rather than a replacement for them, focused specifically on catching the moment an AI application's answers quietly stop matching their sources after a change. Installing the base tool requires no extra infrastructure beyond Python, since capturing test runs, the basic judge, the comparison logic, and a small web viewer for browsing results are all built with the standard library. Support for Claude as a judge, and an on device local judge, are added only if you choose to install those extra pieces.