prove-ai/proveai-sdk

ProveAI SDK is a Python testing framework for multi-agent AI pipelines, the kind of setup where several AI agents each do a small step, like classifying a support ticket, drafting a reply, then checking that reply for quality. The core idea is snapshot testing borrowed from web development: you run your pipeline once, let ProveAI record exactly what each agent received and produced, and pin that as a known good snapshot. From then on, running a verify command re-runs your pipeline and compares the fresh output against that snapshot, grading each agent as unchanged, drifted, or regressed, and failing the check if anything got worse. Getting started takes a few steps: install the package, add a monitor decorator to three to five of your agent functions, capture one real run wrapped in a Trace object, then run a command that pins the captured behavior as a snapshot. After that, whenever you edit a prompt or swap a model, running the verify command shows exactly what changed and whether it flowed downstream to other agents. The tool can also generate a ready made continuous integration workflow file, so this check runs automatically on every pull request, and in that automated setting it only re-checks the snapshots for agents whose underlying code the pull request actually touched, saving time on larger pipelines. When an agent's output changes in a way that is not simply pass or fail, ProveAI uses a separate AI model as a judge to decide whether the change is an equivalent rewording or an actual regression, and caches that judgment so repeated runs do not cost anything extra. There is also an older, still supported workflow for larger scale regression testing across many test cases at once, plus a pytest plugin for teams that prefer to run these checks inside their normal test suite. The project ships with built in support for OpenAI and Anthropic models, and needs extra configuration, such as an API key and endpoint URL, to work with any other provider. The full README is longer than what was shown.