adewale/skill-eval-harness

Skill Eval Harness is a command-line tool for testing whether a custom instruction file called a "skill" actually changes what an AI coding agent does. The core question it answers: when you give the agent a skill versus not giving it that skill, on the exact same task, what changed? To use it, you write a benchmark manifest file listing test cases. Each case has a prompt (the task or question), expected behavior, and assertions defining how to automatically check the output. The harness generates two versions of each case: one where the agent runs with the skill active, and one without. You then run those tasks through whichever AI agent you are testing (the harness has built-in support for Pi, Claude Code, and Jetty), and save the outputs to a folder layout the harness expects. Once the agent outputs are saved, the harness grades them using deterministic checks: does the output contain a certain string, match a regex, include a specific file, or have a JSON field set to an expected value? Optionally, a "judge" assertion can invoke an external evaluation command for qualitative scoring. The harness does not pick a model for that judge step, you supply the command yourself. The resulting report flags a range of outcome types: cases where the skill made no detectable difference ("no lift"), cases where repeated runs gave inconsistent results ("flaky"), assertions that always pass regardless of output ("saturated"), and cases where the agent with the skill still failed. A leakage lint step checks whether the task preparation accidentally exposed the expected answer to the agent, which would invalidate the test. There is also a static HTML review page for browsing outputs and assertion evidence side by side. The tool is installed as a Python package via the uv package manager and is released under the MIT license.