tencent-hunyuan/planningbench

PlanningBench is a research benchmark from Tencent Hunyuan and Renmin University of China that tests how well large language models handle complex planning problems with multiple constraints, priorities, and dependencies. Rather than asking simple questions with obvious right answers, it tests whether a model can produce a complete, executable plan that satisfies all the stated constraints at once. The benchmark releases 467 synthetic planning instances intended for evaluation rather than training. Each instance contains a planning question and a verification checklist that makes it possible to check automatically whether a model's answer satisfies every constraint and objective. The questions are self-contained, meaning everything needed to solve and verify the plan is included in the problem statement itself. The 467 instances span six broad families of planning problems: scheduling and timetabling (fitting tasks into time windows without conflicts), project and production operations (managing milestones and dependencies), routing and travel (coordinating movement across locations), emergency response and public service (allocating resources under urgency and priority), allocation and matching (assigning resources under compatibility and capacity limits), and shift and workforce scheduling (covering all required work while respecting fairness rules). Together these cover more than 30 concrete task types. The data was built through an automated synthesis pipeline: planning scenarios were abstracted into taxonomies of tasks and constraints, candidate problems were generated from those taxonomies, a model attempted to solve them, and a verification step checked the answers. Human reviewers participated in quality control. Difficulty was adjusted by tightening constraints, increasing resource scarcity, and adding dependencies. The dataset is available on Hugging Face and accompanied by a paper on arXiv. The repository contains the evaluation dataset, documentation, and figures. A separate license file governs its use.