ahnafyy/skills-evals

skills-evals is a command line tool that checks whether the instructions you give an AI coding agent are still working. If you write skills, custom agents, rules, or prompt files for tools like Claude Code, GitHub Copilot, or Cursor, this tool tests whether those artifacts still trigger correctly and behave as intended, especially after your codebase or the instructions themselves change. It scans your repository for these files automatically, covering formats like Claude and Copilot skill files, Copilot instructions and custom agents, Claude custom agents, Cursor rules, and prompt files. It has no external dependencies and needs Node version 18.17 or newer to run. The checks run in three tiers. The first tier is structural, checking things like file formatting and description limits, and runs for free in continuous integration. The second tier tests whether realistic prompts actually route to the right skill or agent, also for free, by ranking how well descriptions match sample prompts. The third tier is behavioral: it has a real AI agent follow the instructions and checks, using a grading model, whether the agent actually did the right thing. This tier is meant to run on a schedule, like nightly or weekly, using a cheap and fast model. A key feature is regression detection. You save a baseline snapshot of test results, and on every later run the tool compares against it. If a previously working trigger suddenly fails on a file that was not changed, that signals another skill is now stealing its triggers, and the tool flags it as an error so your team catches the problem in a pull request rather than in production. The project also ships an installable setup skill that can walk another AI agent through configuring skills-evals in any repository automatically, including writing test cases and wiring up continuous integration. The project describes itself as compatible with the eval case format used by Anthropic's own skill creation tooling.