amap-cvlab/abot-world

ABot-World is a research project that turns a single high end desktop GPU into a real time interactive world simulator. Instead of watching a fixed length AI generated video, you can steer the simulation with your own actions, similar to controlling a character in a video game, and the world keeps generating new scenes as you go rather than looping or ending after a set amount of footage. The system runs at 720p resolution and 16 frames per second on a single NVIDIA RTX 5090 desktop GPU, with about 1.2 seconds of latency and using roughly 19 gigabytes of GPU memory. It responds to user actions in real time, so exploring the generated world feels continuous rather than like watching a passive clip. A training method the authors call LongForcing lets the simulation introduce new scenes and dynamics as it rolls forward, so it does not get stuck repeating the same environment or need the user to manually switch prompts to see something new. The project provides a smaller causal model, inference code, a local demo built with Gradio, and an online playground called ABot World Studio where people can try the system without installing anything. The team has also released technical reports and is making a 500 hour video training dataset with action annotations available. Setting it up locally is aimed at people comfortable with machine learning environments. The documented setup was tested on Ubuntu 22.04 with CUDA 12.8, Python 3.12, and an RTX 5090, and installation involves cloning the repository, creating a conda environment, installing a specific version of PyTorch built for that CUDA version, and building two performance libraries called FlashAttention and SageAttention from source, which can involve compiling code and working around glibc version issues on some systems.