robbyant/lingbot-vla-v2

LingBot-VLA 2.0 is an AI model built to help robots understand what they see, what they are told to do, and how to move their bodies to actually do it. This kind of model is often called a vision language action model, because it takes in a camera image and a text instruction and outputs the physical actions a robot should take. The project is a research release from a company continuing earlier work on the same kind of model, aiming to move it from research toward more reliable use on real robots. Compared to the earlier version, this release was trained on a much larger and more varied set of data, roughly sixty thousand hours combining real robot recordings from twenty different robot setups and video recorded from a human's own point of view while performing tasks with their hands. It also expands what kinds of robot parts the model can control, beyond just robotic arms to include grippers, dexterous hands, waists, heads, and mobile bases, by mapping all of these different robot bodies into one shared numerical format the model can reason about consistently. The model additionally tries to predict what will happen next in a scene as part of its training, using two other AI systems to supply visual and depth based hints about how the world tends to change over time. Using this project requires real technical setup: a Python environment manager, a specific version of Python and PyTorch, and a provided setup script that installs the needed libraries, including a specialized library for fast attention computation. The pretrained model weights are downloadable, but training or fine-tuning it further on your own robot data requires several additional pretrained components and a defined data preparation process, including formatting your dataset, mapping your robot's specific parts to the model's shared format, and computing statistics used to normalize the data. The project includes an example of retraining the model on a public robotics benchmark dataset. This is a research project intended for people already working in robotics or machine learning, not a beginner-friendly tool, and it is released under the Apache 2.0 license.