davidspiegs/adtech-eval-lab

This project builds a small test suite for checking how well AI agents handle the messy, real world work of digital advertising operations, often called Ad Ops. The author noticed that while there is a lot of AI research on general office tasks, almost nothing exists that tests AI on the specific kind of work done by people who manage online ad campaigns, so they built two realistic tasks based on their own experience working at ad technology companies. The two tasks ask an AI agent to reconcile billing numbers between an ad server and a supply platform, read PDF invoices, match up inconsistently named campaigns and advertisers, apply specific business rules, and produce a finished Excel report. One task is modeled on a monthly finance close where revenue numbers need to be checked and audited. The other is modeled on investigating a complaint from an advertiser about missing ad impressions or other campaign problems. Both tasks are checked automatically using a set of deterministic pass or fail rules built into the project, so results do not depend on human judgment. The author tested a handful of AI models on these tasks with a limited number of trials, since the project was built on a personal budget. One model passed every trial on both tasks, another model did well but occasionally slipped on one specific calculation step, and a third, cheaper model struggled more, especially with picking the right metrics and handling time zones correctly. The author is upfront that this is a small, synthetic dataset meant to explore how to design fair AI evaluation tasks, not a large scale industry benchmark. The project uses the Harbor framework for structuring and running these evaluation tasks, along with Docker to run each task in its own environment. It includes the test data, the correct reference solutions, the automated checking scripts, and written reports explaining what the author learned while designing the tasks. All the data used is synthetic and made up, based on the author's own industry knowledge, and contains no real company or customer information.