goekmenai/local-llm-matrix

Local LLM Matrix is an app for comparing AI language models you have installed locally through Ollama, a popular tool for running AI on your own computer. The problem it addresses is that online benchmarks measure performance on reference hardware, while the same model may run much faster or slower on your specific machine, with your specific setup and quantization settings. The app runs a set of task-oriented prompts against any models you have installed, measures their speed, evaluates their output, and saves the results. An optional local Judge feature lets you pick one of your installed models to score the responses of the others, keeping the entire evaluation process on your own machine. Results are saved as JSON files, and a SQLite history index tracks performance over time so you can see whether a model consistently outperforms another or whether one result was an outlier. The interface has seven sections: an Overview showing current evidence and recommendations, Models and Sources separating local models from public reference data, a Test section for setting up benchmark runs, a Results section that explains confidence levels and skipped evaluations, a History section for trend analysis, an Analysis and Export section, and a Help section. A stated design principle is that public benchmark numbers and model cards are shown as context only and are never presented as measurements of your local model. The app does not download models automatically, you choose which models to pull in Ollama separately. The app supports German and English interfaces, dark and light themes, and responsive layouts. Results can be exported as Markdown, HTML, CSV, JSON, or ZIP. Setup on macOS involves double-clicking two provided scripts, Linux and Windows require a manual developer start. Built with Python and Streamlit. Apache 2.0 license.