fudancvl/avtrack

AVTrack is a research dataset and accompanying baseline system for teaching AI to figure out who is speaking in videos, specifically in complex, real world scenes with crowds, overlapping voices, and camera movement. Published at ICML 2026, it addresses a gap in existing benchmarks that rely on clean, single speaker, laboratory style footage. The dataset contains 871 videos averaging about 54 seconds each, drawn from interviews, films, anime, operas, narrations, and stage performances. Every video includes pixel level outlines, called instance masks, that track individual people across frames, alongside synchronized audio. Videos are labeled with challenge attributes such as camera motion, occlusion, position changes, and overlapping speech, making it possible to measure how different systems handle specific difficulties. The project also ships a training free baseline pipeline called AVTracker. It combines a visual tracking model called SAM3 with a speech recognition model called Whisper to identify who is on screen and what is being said, then uses a vision language model to match speech segments to the correct person. Optional steps include speech separation through Mossformer2 to untangle overlapping voices, and face recognition through DeepFace as an alternative matching strategy. The code is written in Python and uses a tool called uv for dependency management, with a pinned lockfile for reproducibility. Models are downloaded automatically from Hugging Face on first use, and the dataset itself is also hosted there. The project is released under the MIT License and is aimed at computer vision and audio researchers working on multimodal scene understanding and tracking.