unix-ai-lab/worldreasonbench

WorldReasonBench is a research benchmark that tests whether AI video generators can genuinely reason about how the world changes over time, rather than simply producing visually convincing footage. The central question it asks is: given a starting scene and an action, does the generated video show a future state that is physically, socially, logically, and informationally consistent with reality? The benchmark contains 436 carefully selected test cases with structured question-and-answer annotations covering four reasoning dimensions and 22 subcategories, including world knowledge, human-centric behavior, logic reasoning, and information-based inference. Alongside the main benchmark, the project also releases WorldRewardBench, a companion preference benchmark with approximately 6,000 expert-annotated pairs across over 1,400 videos, designed to evaluate reward models used to score video quality. Evaluation uses two scoring approaches. The first, called Score_PR, combines question-answer accuracy with a dynamic-phase penalty, and the README reports that this metric reproduces human preference rankings with very low rank displacement. The second, S(v), is a weighted score across reasoning quality, temporal consistency, and visual aesthetics. Results from 11 evaluated generators are published in a leaderboard in the README, separated into closed-source and open-source models. The benchmark is positioned as a research contribution to the question of whether video generation systems are becoming true world simulators. It is written in Python, and the dataset is available on Hugging Face. The full README is longer than what was provided.