benchflow-ai/skillsbench-trajectories

SkillsBench Trajectories is a data repository that stores the results and supporting data behind the SkillsBench AI leaderboard, a website that ranks how well different AI models perform practical tasks. Think of it as the public filing cabinet for a benchmarking project. Instead of just trusting a leaderboard's scores, this repository lets anyone peek behind the curtain to see the underlying data and verify how the AI models were evaluated. The repository serves as a bridge between raw experiment data and the live leaderboard website. The actual heavy lifting, storing the detailed "trajectories," which are essentially step-by-step recordings of the AI working through each task, happens on a separate platform called HuggingFace. This GitHub repository holds a processed summary file that the website reads to display scores and pass/fail counts on its task pages. The latest snapshot of data covers 18 different AI model setups running 87 distinct tasks under two conditions: one where the AI had access to curated "Skills" (likely special instructions or tools) and one where it did not, with each scenario tested multiple times. This project is primarily for AI researchers, developers, and evaluators who want to understand how today's top AI models handle complex, multi-step tasks. For example, if a product manager is deciding which AI model to integrate into a new software tool, they could use the live leaderboard and this underlying data to see which model consistently succeeds at tasks similar to their use case. It also serves the open-source community by providing transparent, reproducible benchmarks rather than just marketing claims from AI companies. A notable aspect of how this is organized is the separation of concerns between platforms. The detailed recordings of the AI's problem-solving process live on HuggingFace, which is built for hosting large datasets, while this repository handles the lighter lifting of powering the website. Older experimental runs from individual contributors are still kept here in named folders for historical reference, but the project has clearly shifted its current source of truth to the HuggingFace dataset.