caseclose/cma-harness

CMA-Harness is a research project from Peking University and Tencent's WeChat Vision team about building an AI agent that can understand, generate, and edit images and other visual content over long conversations. The core idea is that instead of trying to keep every image from a conversation in the AI's working memory at once, the system compresses past images into short descriptions, tags, and thumbnails. It then pulls back only the specific past images that matter for the current question, which the authors call selective retrieval. The project names its two main pieces: an Episodic Visual Memory that stores this compressed history, and a Multimodal Executive Controller that decides whether a new message needs the AI to understand something, generate a new image, edit an existing one, or combine several images together, then sends the task to the right process. Together these form what the authors call a cognitive structured multimodal agent, with CMA-Harness being the deployable version that adds tool support for real multi-session use. Alongside the agent, the authors built a benchmark called M2CA-Bench to test how well systems like this handle long conversations involving images. It contains 100 sessions of 20 turns each, covering 55 topics across 8 subject areas, with four levels of difficulty. The benchmark deliberately includes near duplicate images that differ only in small details like color or lighting, so that a system cannot cheat by guessing rather than actually retrieving the right image. According to the reported results, the system retrieves the correct past image around 91 percent of the time across full sessions, drops slightly to about 89 percent on longer sessions, and reaches 82 percent on the hardest cases. It also runs roughly twice as fast per turn compared to a simpler approach that tries to keep the entire conversation in context at once. The README notes that the code and the M2CA-Bench dataset will be released, meaning the actual software may not be fully available yet at the time this was written. There is an accompanying academic paper describing the method and results in full technical detail.