joow0n-kim/collabvr

CollabVR is an AI research framework that solves a mismatch between two types of AI models: Vision-Language Models (VLMs), which are good at logical reasoning but struggle to simulate what things should look like, and Video Generation Models (VGMs), which can produce realistic short video clips but cannot reason about what to do next. On their own, each type fails at multi-step goal-directed video tasks: VLMs lose track over long sequences (called long-horizon drift), and VGMs make simulation errors mid-clip that corrupt everything that follows. CollabVR couples them in a closed loop. At each step, the VLM decides the next action to take, the VGM renders a short video clip for that action, and the VLM then inspects the clip and either accepts it or rejects it with a diagnosis of what went wrong. Two recovery modules then act on that diagnosis. The first, Progressive Planning, controls how many sub-steps to plan at once: simple atomic actions stay as single steps, while more complex tasks get broken into finer steps only when needed. The second, Verification and Re-generation, replays the clip with an updated prompt when something looks wrong, retrying up to a set budget, if all retries fail, it routes to a recovery strategy matched to the diagnosed failure type. The framework is a research project published as an arXiv paper in 2026. Planned video generation backends include Veo 3.1 and VBVR-Wan2.2, and planned evaluation benchmarks include Gen-ViRe and VBVR-Bench. At the time of the README, the code was still being prepared for public release.