mechanize-work/gba-eval

GBA Eval is a benchmark that tests whether AI coding agents can build a working Game Boy Advance emulator from scratch in just 24 hours. The project comes from Mechanize, a company that builds evaluation environments for training and testing large language models, and this repo is a public example of that kind of work. The idea is simple: give an AI agent one day to write a software emulator with WebAssembly support, then grade the result against Mesen2, a well known and highly accurate GBA emulator, used here as the reference. Nine emulators built by different AI models during a May 2026 leaderboard run are published in a companion repository, each with its full commit history and a transcript of the chat log that produced it, so anyone curious can see how the agent worked step by step. Grading is split into three weighted sections. Gameplay replays, which check real button press input and end to end behavior, make up 60 percent of the score. Procedural tests covering the CPU, memory, timers, and DMA account for 20 percent. Audio accuracy, measured by comparing sound output frame by frame against the reference emulator, makes up the final 20 percent. These weights can be changed in a configuration file if someone wants to run the benchmark differently. To actually run a grading pass, Docker is the only required dependency, along with git submodules and git LFS to pull in the reference data and test material. The grading engine itself is written in Rust and can precompute reference data once, then score any candidate wasm file against it. Users can also rebuild the reference emulator from source if they do not trust the bundled version. Licensing is mixed rather than a single blanket license. The project's own code, including the specification, test corpus, and harness, is MIT licensed. Code tied to the Mesen2 reference emulator, including the build script and the compiled reference binary, is GPL-3.0, since it derives from that upstream project. Test ROMs carry their own individual licenses depending on where they came from.