zefulin/worldpilot

This repository contains the code for a research project called World Pilot, which is about improving how AI systems control robots. The core idea involves a class of AI models that take in visual and text descriptions of a scene and output robot movements. These are called vision-language-action models, and they are used in robotics to let a robot understand a situation and decide what to do next. World Pilot improves on those models by adding two extra sources of guidance, both coming from a separate component called a World-Action Model. The first type of guidance is called Latent Steering: it generates a prediction of how the current scene is likely to change, and feeds that prediction into the main model's internal reasoning so the model has a sense of where things are heading. The second type is called Action Steering: it produces a high-level motion hint, essentially a rough idea of the trajectory the robot should follow, and passes that to the part of the model that generates the actual movement commands. The result is that the robot's decision-making gets three inputs at once: the normal visual and language understanding of the current scene, an anticipated view of how the scene will evolve, and a motion direction hint. According to the paper, this combination reaches state-of-the-art performance on a standard robot manipulation benchmark called LIBERO-Plus and on real-robot tests. The repository includes code for training the model from scratch and for running evaluations. Pretrained model weights and a precomputed dataset cache are available on Hugging Face for people who want to test the system without training it themselves. Documentation covering environment setup, training steps, and evaluation procedures is split into separate guide files linked from the README. This is academic research code released alongside the paper, which is posted on arXiv. It is intended for researchers working on robot learning and AI-controlled manipulation tasks.