outsourc-e/bench-loop

BenchLoop is a command-line tool for benchmarking AI language models running on your own computer, not on a cloud server. When people run large language models locally on their own hardware using tools like Ollama or LM Studio, they often want to know how fast is it, how accurate is it, and can it actually perform useful tasks reliably. BenchLoop provides a repeatable, structured answer. It runs seven test suites against a local model: speed, measuring how fast the model generates text, tool call correctness, whether the model can correctly invoke external functions, coding ability using executable Python tasks, data extraction, instruction following, reasoning and math, and a full multi turn agent loop where the model calls tools, receives results, and works step by step toward a goal. After running, BenchLoop gives you a numerical score broken down across these categories. Every run is saved to your disk, and completed runs are automatically shared to a public leaderboard at bench-loop.com so you can compare your hardware's results with others. The tool works with any model server that uses the OpenAI API format or Ollama's API, including LM Studio, vLLM, and others. It also ships with a local web dashboard, built with FastAPI and React, for visualizing and comparing your past benchmark runs. No account or API key is required. You would use BenchLoop when choosing which local AI model and hardware combination gives the best real-world performance for your specific use case. It is written in Python and available via pip or pipx.