redmadrobot-rnd/harness-bench

Harness Bench is a framework for benchmarking AI coding agents and assistants, often called harnesses. It runs a chosen agent against sets of predefined tasks, scores how well the agent's answers perform, and shows the results in a web dashboard that keeps a history of past runs. The project supports three specific agent harnesses out of the box, named Hermes, OpenClaw, and OpenCode, and also lets users write a custom benchmark by creating a single Python class, which can then be run against any supported agent. The framework ships with several built in benchmarks covering different skills: factual question answering judged by another AI model, function calling ability based on the Berkeley Function Calling Leaderboard, a multi turn version of that same test, code generation checked by running real unit tests, long term memory retention, finding a specific fact hidden inside a large amount of text, fixing real GitHub issues inside a sandboxed environment, a multilingual version of that GitHub issue benchmark, and a benchmark simulating tasks inside a fictional company. To run it locally, users need Python 3.12 or newer and Docker Desktop, since the AI harnesses run inside Docker containers for isolation. PostgreSQL is optional, without it, run history is simply not saved. After installing the package and copying an example configuration file with an API key and model name filled in, users can either start a local web service to configure and launch runs from a browser, or run everything from the command line without a UI. A Docker Compose setup is also provided for running the whole system as a deployed service, with its own database configuration separate from any individual run's settings. The command line tool can serve the web interface, run benchmarks directly, list past runs, show and compare results between two runs, and manage the database. Documentation is mirrored in both Russian and English inside the repository's docs folder.