heartune/robotheory-79k

ROBOTheory-79k is a large benchmark dataset designed to test how well AI language models understand robotics engineering theory. The central question it asks is whether modern AI systems genuinely grasp robotics concepts at a deep level, or whether they are simply good at pattern-matching based on text they have seen during training. The dataset contains 79,239 expert-level questions across four major areas: mathematical foundations, mechanical systems, perception and control, and electrical systems and programming. These are divided into 24 sub-fields. Questions are available in Chinese, English, and French. Each question comes with a step-by-step solution verified by domain experts, so the benchmark can also be used to train or fine-tune AI models, not just evaluate them. The evaluation suite built on top of this dataset, called ROBOTheory-Bench, uses a 30% sample of the full dataset that preserves the distribution of question types and subject areas. The paper accompanying this project tested 24 of the best-known AI language models in a zero-shot setting, meaning the models were given no worked examples before being asked questions. The best-performing model scored 64.78%, while human experts scored 83.4%, leaving a gap of nearly 19 percentage points. The results also showed that all models performed notably worse on open-ended calculation and reasoning questions than on multiple-choice questions, suggesting that probabilistic pattern-matching is part of what these models rely on. This repository provides the evaluation scripts, prompt templates, and judge configuration needed to reproduce the paper's results or to evaluate a new model against the benchmark. The dataset itself and a companion academic paper are linked from the project's website. Both were still pending public release at the time this snapshot was taken.