ant-research/memdreamer

MemDreamer is a research project from Ant Group and Zhejiang University focused on helping AI systems understand long videos. Instead of trying to process an entire video at once, it separates two jobs: noticing what happens in the video, which is perception, and figuring out answers to questions about it, which is reasoning. The core idea is a Hierarchical Graph Memory. As a video streams in, the system builds a structure that goes from a single video root down through super events, macro events, and finally individual entities and events. This lets the system keep track of a long video without holding every detail in memory at once. To answer a question about the video, MemDreamer does not just search once. It uses an agentic, tool augmented retrieval process built around an Observation, Reason, Action loop. Three categories of tools support this: hierarchical navigation, precise search, and graph traversal. The system observes what it currently knows, reasons about what it still needs, and takes an action such as searching deeper into the memory graph, repeating this cycle until it can answer. According to the README, this approach reaches state of the art results across four video understanding benchmarks, LVBench, LongVideoBench, Video-MME, and EgoSchema, closing the gap to human expert performance to about 3.7 points. As of the README, the project has released the agentic retrieval code along with precomputed memory files and inference trajectories, available on HuggingFace. The code for actually building the memory graphs from raw video has not been released yet. Getting started involves cloning the repository, installing Python dependencies, and downloading the prebuilt memory and trajectory data to reproduce results, or setting up custom embedding servers to run retrieval with your own LLM backend. This is written in Python and comes from an academic research team, accompanied by a paper on arXiv. It is licensed under Apache 2.0. Because memory construction code is still pending and the setup involves downloading large precomputed datasets plus configuring embedding servers, this is best suited to researchers and developers already comfortable with video understanding pipelines rather than someone looking for a simple drop in tool.