antoinef23/vasari

Vasari is a tool for evaluating how well an AI feature, such as a chatbot, an autonomous agent, or a system that answers questions using your own documents, is actually performing. It runs entirely in your browser, so you can load conversation logs from almost any AI system, read through them, and mark each one as a pass or a fail with a short comment explaining why. The approach follows a method popularized by AI evaluation researcher Hamel Husain: look closely at real conversations, label them by hand, and only automate that judgment with an AI model once you have proven the AI agrees with human reviewers closely enough to trust it. Vasari supports this whole process. After labeling conversations, you can tag why each failure happened using short free text notes, and the tool will group similar notes into broader failure categories for you. From there, you can build a small AI judge for each failure category and test how well it matches your human labels using a confusion matrix and standard accuracy measures, including true positive rate, true negative rate, and Cohen's kappa, a statistic that accounts for agreement happening by chance. Vasari can read conversation logs in several common formats, including OpenTelemetry traces, Langfuse exports, and simple lists of chat messages, and displays them as a readable conversation timeline rather than raw JSON. It also includes a built in tool to redact personal information like emails, phone numbers, and names before conversations are reviewed or sent to an AI judge. Everything runs locally in your browser with no backend and no account needed, and the only network calls it makes are ones you explicitly trigger, such as calling an AI judge model. It can be exported as a single self contained web page that works offline. The project is written in TypeScript, is released under the MIT license, and currently has 1 star on GitHub.