tencent/vita

VITA-E is a project from Tencent that lets a robot see, hear, speak, and act at the same time, much like a person would. Instead of waiting to finish one task before listening for a new command, a robot using this system can respond to your voice while it is actively doing something else. You can even interrupt it mid-action, tell it to stop, or ask it to switch tasks, and it will adjust smoothly and naturally. Under the hood, it achieves this using a dual-model setup: one model acts as the "doer" executing the current task, while a second "listener" model stays alert for new voice commands. When you speak, the system uses a vision-language model (an AI that understands both images and text) to process what you said and generate special control tokens. These tokens act as direct system commands, allowing the AI to immediately manage and redirect what the robot is doing without waiting for a full task to complete. This would be useful for robotics engineers or researchers building physical humanoid robots that need to interact safely and naturally with humans. For example, if a robot is picking up an object and you suddenly say "stop" or "switch to the other box," the robot can halt or change course almost instantly. The creators tested this on a physical humanoid robot and reported high success rates for task switching, emergency stops, and speech interruptions, with an average voice response time of just over two seconds. The project is built by combining two existing AI frameworks: VITA-1.5, which handles the voice and vision understanding, and NVIDIA's Isaac-GR00T, which handles the actual physical movements. The training happens in two stages, first teaching the AI to understand commands, then fine-tuning the entire system end-to-end. Notably, the project includes a web demo that lets you experience the conversational interaction using simulated robot data, so you can try it out without needing an actual robot, though you will need a computer with a substantial amount of graphics memory.