dingyue772/omnisift

OmniSIFT is a research project published at ICML 2026 that tackles a practical problem in AI: large language models that can understand both video and audio at the same time, called omni-modal models, process a very large number of data tokens, making them slow and expensive to run. OmniSIFT is a compression technique that trims down the number of tokens the model has to handle without significantly hurting its ability to understand what is happening in an audio-video clip. The key insight is treating video and audio differently. First, video frames are analyzed and redundant patches are removed, patches that look similar to nearby frames or to other patches within the same frame. This produces a smaller set of important video anchor tokens. Those anchors are then used to decide which audio tokens to keep: audio that aligns with what is visually important is preserved, while unrelated audio is discarded. This two stage process, video first, then audio guided by video, is what the authors mean by modality asymmetric compression. The codebase provides a pre-trained model checkpoint called OmniSIFT-7B available on Hugging Face, and includes integration code for the Qwen2.5-Omni model family. A quick-start example in the README shows how to load the model, configure how aggressively each modality is compressed using two ratio parameters, and run inference on a video file to get a text description. The project is written in Python and released under the Apache 2.0 license.