doric-builder/plotline

Plotline is a benchmark that tests whether an AI language model can keep track of a long, messy conversation over time, not just remember things inside a single exchange. Each test scenario plants a set of facts across a 13-turn working session, then tries to trip the model up with tangents, attempts to reverse decisions that were already locked in, flattery meant to sway an answer, precision traps, fake real-world name lookalikes, mid-session corrections to earlier facts, two projects discussed at once, pressure from a supposed authority figure, and questions that only make sense if the model checked its own record first. Each run is scored on ten different axes, and the most important one measures what the model does when it does not clearly have a fact: does it admit that, or does it make something up. The project ships four ready-made scenarios: a restaurant-tech startup, a healthcare operations rollout, an independent music album launch, and a two-project infrastructure retrofit. Each one is built to stress a different failure mode, such as facts changing mid-session, or two topics bleeding into each other under pressure. The project is upfront that one of the four scenarios was used while tuning the developer's own product, so scores on that one carry a caveat, while the other three were written without ever being tested against that product first. To run it, you point the tool at a scenario using an API key from Anthropic, OpenAI, or Gemini, or you write a small turn module that wraps your own AI system so it can be tested the same way. Results are written out as transcripts, which you then score using the included rubric and a methodology guide that recommends running each scenario at least three times and reporting a range of outcomes rather than a single number. The scenario texts, rubric, and methodology are released under a Creative Commons license, while the runner code itself uses the MIT license.