tanisha-byte/roger-that

Roger That is a testing tool for AI voice and chat agents that uses real air traffic control radio procedures as its exam. In real aviation, every instruction a controller gives has a required, exact readback, so a pilot cannot just say something that sounds close, it has to match specific rules. This project takes that same strict structure and uses it to grade AI agents without needing another AI to act as the judge. Instead, a parser checks the agent's response against the rules directly, which makes scoring repeatable and fair. The project includes six scenario types, such as a descent instruction, a rapid sequence of commands, a callsign mix-up trap, a case where the agent mishears something, and an emergency situation, and each one is randomized with a seed so an agent cannot just memorize answers. A scorer module reads the transcript, converts spoken numbers and the phonetic alphabet into a standard form, checks whether the agent caught any traps or made any hearing errors, and produces a weighted score. The scorer agrees with human graders on a hand-labeled test set at least 99% of the time. You can run this two ways: place a real outbound phone call to a live Bolna voice agent and grade the actual conversation, or run a text-based campaign against any OpenAI-compatible language model. Results are stored in a database and viewable through a dashboard with two views, one that replays a session as a pixel-art radar animation showing successes and mistakes, and one that shows a leaderboard, learning curves over time, and full transcripts. Setup involves creating a Python virtual environment, installing dependencies, and running the test suite before starting the dashboard with uvicorn. Live phone testing needs a Bolna API key, and testing language models directly needs an OpenAI-compatible API key. The project also includes an experimental learning-loop feature for helping agents improve between sessions, though the README notes this part has not been tested against a live system yet.