hengyuan-hu/via

VIA stands for Visual Interface Agent for Robot Control. It lets a general purpose AI coding agent, such as Claude Code or Codex, control a simulated robot through a browser based 3D interface. The agent interacts with the simulated robot using MCP tools, which is a standard way for AI assistants to call external functions and interfaces. The project relies on a robot simulation framework called LIBERO, which is included as a linked submodule rather than being copied directly into the repository. Setup involves cloning the repository, pulling in the LIBERO submodule, and using a tool called uv to create a Python 3.10 virtual environment and install all dependencies, including a one time browser install step for the MCP server. The README notes that torch and torchvision are pulled from PyTorch's own package indexes, so the same dependency lock file works whether you are running on a CUDA enabled Linux machine or a Mac using its built in GPU support. To use VIA, you first source a setup script that activates the virtual environment and sets the right Python paths for both the main project and the LIBERO submodule. From there, you start a simulation recording process in one terminal for a chosen task, then launch either Claude or Codex in a second terminal to control it. There is also an evaluation mode that can run the same task across multiple random seeds at once, in parallel, to test how consistently the agent performs. The MCP server that connects the AI agent to the simulation starts automatically in the background and does not need to be run manually. The repository is organized into a few clear parts: the main code package that handles simulation recording, evaluation, and the MCP server, a folder of per task instruction prompts used during evaluation, the setup and dependency files, and the LIBERO submodule itself. The README does not state a license for this project.