pierreolivierbonin/verbatimeter

Verbatimeter is a small Python tool that checks how much of an AI generated answer actually comes word for word from the source text it was supposed to be grounded in. It is aimed at teams building RAG systems, retrieval augmented generation, where a model pulls in context and is expected to answer using that context rather than making things up. The tool works by comparing an answer against a source and reporting which words were copied directly, shown in green, and which words the model added on its own, shown in red. Two ways of measuring this are offered. The default, verbatim reuse, looks for contiguous runs of at least three matching words in a row, since single matching words are too easy to happen by chance. A second mode, verbatim paraphrasing, uses a longest common subsequence comparison to catch cases where the source's words are reused but rearranged or split apart. There is also a quotation mode that checks only the text inside quotation marks, which is useful for catching a model that claims to be quoting a source but actually invented the quote. Verbatimeter can be used in three ways: as a Python decorator wrapped around a generation function so every answer is checked automatically, as a command line tool that takes source and answer text or files, or as a gate in a continuous integration pipeline that fails the build when grounding drops too low. The checking itself is fully deterministic and does not rely on another AI model to judge the answer, so results are repeatable. The tool supports any language that separates words with spaces, including French, Hindi, Urdu, and Arabic, though it does not yet handle languages without spaces between words, such as Chinese, Japanese, or Thai. It has a single runtime dependency, tiktoken, and needs no network access once installed. Extracting text from PDFs or other document formats is left to other tools, verbatimeter only works on plain text you already have.