opensquilla/claw-swe-bench

SWE-bench is a collection of real GitHub issues that AI coding agents attempt to solve by writing code patches. Researchers and developers use it to measure how well different AI agents handle practical programming problems. claw-swe-bench is a tool that makes running multiple different agent frameworks on this benchmark fair and consistent. The core problem this project solves is comparability. When different teams test their AI agents on SWE-bench, they often use different prompts, different ways of collecting results, and different evaluation steps. That makes it hard to know whether one agent truly outperforms another, or whether the difference comes from the surrounding test setup. claw-swe-bench pins those surrounding details so only the agent itself changes between runs. It does this through a shared adapter layer. Each supported agent framework, called a "claw", gets a small adapter file that plugs it into the same orchestration system. Every claw receives the same prompt text, runs inside the same Docker container environment with the same resource limits, and has its code changes collected the same way once it finishes. Adding support for a new agent framework means writing one new file and registering it, rather than rebuilding the whole evaluation pipeline. The project ships with adapters for five agent systems: openclaw, hermes, nanobot, zeroclaw, and generic. Each one operates differently under the hood, but the outer shell stays identical. The benchmark covers 350 coding problems drawn from real GitHub repositories across eight programming languages, with a smaller 80-problem subset for cheaper testing iterations. To run an evaluation, users install their chosen agent framework on a host machine, copy an example config file, fill in API keys, and then call a single Python command. Results land in a folder organized by run ID, including per-problem logs, the code patches the agent produced, and token usage where available. A second command runs official SWE-bench scoring on those patches. The project accompanies a technical report on arXiv that describes the benchmark design, fairness controls, and results across the five supported agents.