kltownsend/loopcontrolbench

loopcontrolbench is a benchmark that tests how well AI coding models can fix real software bugs. Instead of using an AI or a human to judge whether a fix is good, it uses the project's own automated tests as the judge. Each task comes from a real merged bug fix pull request on GitHub: the buggy code before the fix, the maintainer's own test that catches the bug, and the actual fix that was merged. A model only passes a task if the named test flips from failing to passing after its fix, nothing is guessed or inferred. The tasks come from Python libraries with reproducible, maintainer written tests, so this measures a specific and useful slice of real world bug fixing, not a general claim about all software bugs. A reference run using the model gpt-5-mini, with one attempt per task and no retries, solved 46 out of 70 tasks, about 65.7 percent, across projects like h2, pyjwt, dateutil, more-itertools, marshmallow, click, prettytable, and requests. Every task in that run produced a clear pass or fail result, with none excluded due to setup problems. Because the tool downloads and runs code from third party repositories, including their install steps and test suites, this is treated as running untrusted code. The project strongly recommends running it only inside an isolated, disposable container or virtual machine with no cloud credentials, no access to your home directory, and no access to internal networks. The default runner enforces these protections automatically, including blocking network access during tests and limiting what dependency installation can reach. The project publishes the mechanical scoring system itself, meaning the task pool, the checks that admit a task, and the scoring runner, but does not publish opinions about what a score means for any particular AI model or vendor. That judgment is left to whoever runs the benchmark.