m1chaelpeng/perceptionrubrics

PerceptionRubrics is the code and data release for a research paper called "PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception," accepted at ICML 2026. It gives researchers a way to test how well AI models describe images, and it checks those descriptions against very specific, per-image checklists instead of just comparing them loosely to a reference caption. The core idea is a rubric system built from 1,038 densely detailed images paired with over 10,000 individual rubrics. Each rubric falls into one of two buckets: Must Right items, which are facts a caption absolutely has to get correct, and Easy Wrong items, which are common mistakes or hallucinations models tend to make. Scoring uses a gated method rather than a simple average, so getting a mandatory fact wrong causes a sharp penalty instead of being smoothed over by other correct details. The images span seven categories: natural scenes, documents and OCR, digital interfaces like web and mobile UIs, structured data such as charts and tables, STEM and expert diagrams, logic puzzles, and creative or cultural content. The project reports that even strong models struggle to satisfy every rubric at once, that open source models trail proprietary ones by roughly 8 percent in this kind of fine grained perception, and that this rubric based scoring lines up better with human judgment than older benchmarks do. To use it, you clone the repository, set up a Python 3.11 environment, and install the package. A single script runs the full pipeline: generating captions with a chosen vision model, then judging those captions against the rubrics using a separate judge model, with gpt-oss-120b recommended to reproduce the paper's numbers. The steps can also be run separately, first predicting captions and saving them, then scoring those predictions against the dataset's rubrics. Settings like which model to test, which model judges, and how many samples to run are all controlled through environment variables. This is a research evaluation tool rather than an end user product, aimed at people building or benchmarking multimodal AI systems who want a stricter, more human aligned way to measure captioning accuracy. It is released under the Apache 2.0 license.