agi-eval-official/forte

FORTE is a benchmark for testing AI agents on everyday office work. Instead of measuring coding skill alone, it checks how well an agent can handle tasks that people in real jobs do every day, spread across 15 corporate professions such as marketing, sales, finance, HR, legal, and design. Each task comes as a single file describing what to do, along with a small office style environment that can include spreadsheets, Word documents, slide decks, PDFs, CSV files, images, or source code. An AI judge then reviews the finished work and checks it against a list of rubric items written by human experts. A task only counts as a pass if every rubric item is satisfied. The full benchmark has 180 tasks with at least 10 per profession, but this repository ships one demo task for each profession so people can try the setup before running the whole thing. One profession, SRE, is missing its supporting skills because they depend on internal systems the maintainers cannot share. The project also suggests installing general office skills, such as ones for reading and editing Word, PowerPoint, Excel, and PDF files, to help the agent work with these file types more reliably. To run FORTE you need Docker and Python 3.10 or later. The agent itself runs inside a Docker container using a tool called OpenClaw, and the benchmark script builds that container image before running tasks. Setup involves copying example config files, filling in an API gateway URL and key, then building the Docker image and running a script. Each task can be run multiple times, and results are saved as JSON and workspace files so you can inspect exactly what the agent did and how it was judged. Three metrics summarize performance across the three runs per task: an average score, a pass rate, and a strict pass rate requiring every run to succeed. A Chinese translation of the documentation is included.