hershalb/music-video-arena

Music Video Arena is a TypeScript project that pits different AI language models against each other in a contest to make the best music video for a given song. Each model gets the same song, a fixed dollar budget, access to FAL and Replicate video and image generation services, a web search tool, and the ability to run local shell commands including ffmpeg. From there each model works independently: researching which generation models to use, generating video clips, editing them together, watching the result, and refining it until it produces a finished output video synced to the track. Rather than scoring the models with an automated test, the winner is judged subjectively by a human watching the results. The author's stated goal is to observe how different models plan, manage a budget, use tools, recover when something fails, and show taste, things a typical one shot coding benchmark cannot capture. The tool deliberately leaves out any dedicated audio or video editing helper, since part of the experiment is seeing whether a model can figure out on its own how to inspect and assemble media using only ffmpeg. For each run, the project creates an isolated workspace, copies in the song, and runs a multi step loop where the model calls tools until it stops on its own, or hits a step limit or time limit. Every paid generation call is checked against the remaining budget, and once the budget runs out further paid calls are refused, though the model can still keep editing with ffmpeg. Every model message, tool call, result, and cost is logged, and a run produces the final video file, a structured event log, a readable transcript, and a cost summary broken down by model and generation call. Because the models cannot hear the actual audio, an optional lyric transcript with timing can be supplied, generated with a tool like WhisperX, so a model can sync video clips to specific lyrics. Setup requires Node.js, an .env file with API keys, and ffmpeg on the system path. The readme includes clear warnings: only use songs you actually have the rights to use, real money is spent on generation and token costs so budgets should start conservative, and since the tool lets a model run arbitrary shell commands, it should be run in an environment you trust or inside a container.