nienhq/papertrail-bench

PaperTrail-Bench is a benchmark for testing whether an AI system can correctly find and cite old records buried in years of business email, similar to how a bookkeeper would look things up. It is built around a simulated company's multi year email and attachment archive, with questions that have a single provably correct answer and an exact source citation. The project argues that no existing benchmark really covers this kind of task. Other benchmarks either test memory of a chat conversation or test reading a small set of formal financial documents, but nothing tests searching through messy, redundant, real world style business correspondence spanning years. The way it works is that the correct facts and their timeline are written first, as a structured record of business events, including things like amendments and changes over time. The email archive that a system actually searches through is then generated from that structured record, deliberately made messy and repetitive the way real email is. Every benchmark question is computed directly from that structured record, so there is always a checkable right answer along with the exact spans of text that justify it. Questions are split into separate categories that are scored independently, covering things like looking up a specific record, tracing a chain of amendments to a document, asking what was true as of a certain date, combining information across several email threads, telling apart similarly named entities, correctly saying no matching record exists, and citing the exact right source. The project is at an early stage: the tool that generates the fake company data and the format for the correct answers already work, but only three of the seven planned question categories are built, and the part of the project that actually scores an AI system's answers does not exist yet. It requires Python 3.12 or newer and has no other software dependencies to run. It is released under the MIT license.