li-jialu/toolbudgetbench

ToolBudgetBench is a benchmark and demo tool for studying a specific problem in building AI agents: when an agent has access to many tools, such as 20, 50, 100, or 200 of them, someone has to decide how many of those tool descriptions to show the agent at once. Showing every tool uses up a lot of context space, but showing too few risks hiding the exact tool the agent actually needs. This project makes that tradeoff something you can measure with numbers instead of guessing. Running the included demo command shows how a progressive disclosure approach works in practice: the agent first sees a compact list of tool names, then a small set of expanded tool details, and finally a report showing which tool got picked and how many tokens were saved compared to showing everything up front. The project includes a generator for creating test scenarios with fake tool registries and tasks, plus a more realistic fixture with 48 tools and 32 tasks that include multi step tasks, permission checks, and tasks that can be executed and checked for a real outcome rather than just checking which tool name was picked. There is also a larger 200 tool version for stress testing with many distractor tools mixed in. Nineteen different strategies for deciding what to show the agent are included, ranging from showing all tools, to retrieving only the top matches, to routing by category, to an adaptive strategy that blends several approaches. The tool also includes diagnostics that flag weak tool descriptions, missing constraints, tools that overlap in meaning, and text that resembles a prompt injection attempt, along with checks for how often risky or permission requiring tools get exposed unnecessarily. It has no required dependencies beyond the Python standard library, and ships with an installable skill for the Codex coding tool so results shown in the README are useful for reproducible comparisons between strategies, though the authors are careful to note these are proxy results rather than a claim about how real large language models would rank the same strategies. The full README is longer than what was shown.