uipath/coder_eval

Coder Eval is a tool for testing how well AI coding agents like Claude Code, Codex, or Google's Gemini based Antigravity actually perform on real tasks, and for checking whether custom skills built for those agents work the way they are supposed to. Instead of measuring a model's general coding ability against a fixed public benchmark, it runs the agent inside a sandboxed environment against tasks you write yourself in YAML files, then checks the actual files and commands the agent produced against scoring criteria you define. The tool is useful in a few concrete situations. If you built a custom skill for Claude Code, you can use it to confirm the skill actually gets triggered when it should, and keep re-checking that as models and prompts change over time. If you want to compare how Claude Code performs against Codex or Gemini on the same task, or compare two different prompts or model settings side by side, it supports that as an experiment layer. It can also be wired into a continuous integration pipeline so a build fails automatically if an agent's performance or a skill's reliability quietly gets worse. Scoring is not simple pass or fail. Each task can have multiple weighted criteria worth a fraction between zero and one, covering things like whether specific files exist, how similar generated code is to an expected answer, or a rubric judged by a separate language model. The tool also records detailed information about every run, including each tool call the agent made, token counts, and cost, which can be streamed and viewed in real time. Getting started requires Python 3.13 or newer, the uv package manager, and the Claude command line tool. After installing dependencies, a user can validate a task file without spending any tokens, then run it for real and view a report of the results. The project is open source under the Apache 2.0 license and ships with anonymous, opt-out usage telemetry that the documentation states never includes prompts or file contents.