saccharomycetes/activevision

ActiveVision is a benchmark designed to test how well AI systems can reason about visual information through multiple steps, rather than from a single glance. The core idea is that real vision involves an ongoing loop of looking, thinking, and looking again. The benchmark includes 17 tasks built around this principle. The gap it highlights is striking: frontier AI models score around 10% on these tasks, while humans reach 96.1%. The project comes from researchers at the University of Southern California and includes a paper, a website, and a downloadable dataset. The tasks themselves require iterative visual reasoning. One example asks the model to follow a colored strand from a starting letter to its other end, listing every label it encounters along the way. Another asks it to count the distinct regions that worn paths carve across a field. Both examples demand that the viewer trace or build up an answer step by step, which is exactly where current models struggle. The repository provides evaluation scripts for three model families: Claude, GPT, and Gemini. Each script accepts a model identifier and a reasoning effort level, allowing you to control how much thinking the model does before answering. The project also includes an optional ablation where a coding agent can inspect images using code, simulating a tool use scenario. In that setup, the best agent reaches 50.6% accuracy, still well below the human baseline. Results from every run are written to a structured output directory. Each task produces a file of graded records and a summary of per task accuracy, with a combined rollup at the end. Grading works by extracting the model's final answer from a designated answer tag, normalizing it so that formatting differences do not matter, and comparing it against the ground truth by exact match. Runs are resumable, so partial evaluations can be continued without repeating completed items.