saif658/llmstats

LLMstats is a tool that automatically tests and compares 47 AI language models from three free-tier providers, then publishes the results to a live public dashboard that refreshes roughly every three hours. The providers covered are OpenRouter (which routes to models from OpenAI, Meta, NVIDIA, Qwen, Google, and others), Groq, and Mistral. The entire system runs on GitHub infrastructure at no cost. A scheduled job fires every three hours, sends test requests to each of the 47 models, records how fast they respond and whether they succeed, then saves the results. A companion job builds a static website from those results and publishes it to GitHub Pages. There is no server to manage or database to maintain. The live dashboard has five views. The Overview shows summary cards and success-rate trend charts. The Leaderboard ranks models by a composite score and lets you sort by speed, throughput, or reliability, with a chip showing which provider each model comes from. The Explorer lets you drill into a single model to see its response-time history and error breakdown. The Timeline shows the history of each three-hour run. The Compare view lets you pick two models and see them side by side. Anyone can fork the repository and run the same dashboard for their own data. Setup takes fewer than five minutes: fork the repo, add three free API keys as GitHub repository secrets, enable GitHub Pages, and trigger the first benchmark manually. From that point the cron job keeps the data current automatically. The project uses Python for the benchmark runner and plain HTML plus JavaScript for the dashboard. The architecture was originally inspired by a similar project called NIMStats, rebuilt to cover multiple providers side by side.