dabit3/agent-bench

Agent-bench is a tool for putting AI agents through a standardized test suite so you can compare them on what actually matters: did they get the right answer, how much did it cost, and how long did it take? Instead of running ad-hoc prompts and eyeballing results, you run a command and get a clear report showing pass/fail rates, token usage, total cost in dollars, and average response time. The tool ships with 17 built-in tasks across five categories: reasoning (math, logic puzzles), coding (code generation, debugging), data (JSON and CSV parsing), tool-use (function calling), and research (synthesis and explanation). You pick a suite, point it at a model like GPT-4o or Claude, and it runs the tasks, checks the answers, and produces a report. You can save results to a file and then run a compare command across multiple models to see them side by side. You can also write your own custom test suites in a simple YAML file, defining your own prompts and validation rules for what counts as a correct answer. This is for teams building AI-powered products who need to make decisions like "should we switch from GPT-4o to Claude" or "did our latest prompt change break something." A startup deciding between models can run the same suite against both and see a concrete cost-and-accuracy comparison. A team that just updated their agent's system prompt can run benchmarks before and after to catch regressions. It also integrates into CI/CD pipelines, so you can automatically block a pull request if the success rate drops below a threshold. One notable design choice is that it works with any LLM provider through an adapter pattern, meaning you aren't locked into OpenAI or Anthropic. You can even write a custom adapter for your own agent. The project also emphasizes being lightweight with minimal dependencies, which keeps installation and runs fast.