hanyangyu1021/maskwam

MaskWAM is a research project from researchers at HKUST, Tencent Robotics X, and Tsinghua University. It is a system for controlling robot arms in manipulation tasks (picking things up, stacking bowls, folding towels, opening drawers). The paper was released in June 2026, and the code has not yet been made public. The core idea is about how a robot figures out which object to interact with. If you only describe a task in words (pick up the cup), the robot can get confused when there are many similar objects in view. MaskWAM adds a visual hint: a mask, which is a simple outline or silhouette drawn around the target object in the first camera frame. That outline tells the robot exactly what to focus on, even when the scene is cluttered. The model also predicts future masks, not just future video frames. Most robot learning systems predict what the camera will see next. MaskWAM predicts both the next images and the next object outlines at the same time. This forces the system to pay attention to the relevant object rather than the background or nearby distractors. Actions (the actual robot movements) are predicted alongside these visual outputs in one unified process. The researchers tested MaskWAM on two standard simulation benchmarks called LIBERO and RoboTwin 2.0, as well as on physical robot arms. They report that the system handles language-ambiguous tasks (where multiple valid-looking targets exist) better than approaches that rely on words alone. Performance also holds up when distractors are added, lighting changes, or new object instances appear. As of June 2026, the repository contains only the paper announcement and project page links. No training code, inference code, checkpoints, or installation instructions have been released yet. The README lists all of these as planned future releases.