minedojo/voyager

Voyager is a research project that lets an AI agent play Minecraft on its own, learning new skills continuously without a human guiding it. The agent uses GPT-4, a large language model, to read the game environment, decide what to do next, write short programs to carry out actions, and then check whether those actions worked. It keeps building on what it already knows, so over time it becomes capable of doing things it could not do at the start. The system has three main parts. First, an automatic curriculum figures out what the agent should try to do next based on what it has already learned and what the game environment offers. Second, a skill library stores working code snippets for things the agent has successfully done, like crafting tools or navigating terrain. When a new task comes up, the agent can pull relevant skills from that library rather than starting from scratch. Third, an iterative feedback loop lets the agent try an action, see what happened, and revise its code if something went wrong, up to several attempts per task. The published results show the agent collecting 3.3 times more unique item types, traveling 2.3 times farther, and unlocking technology milestones up to 15 times faster than earlier automated Minecraft agents. Importantly, a skill library built in one Minecraft world can be transferred to a brand-new world to solve new tasks, which earlier approaches struggled to do. Setting up Voyager requires Python 3.9 or newer, Node.js 16.13 or newer, an actual Minecraft installation with specific mods, and an OpenAI API key for GPT-4 access. The process involves installing Python packages, Node.js packages, and configuring a local Minecraft instance to accept connections from the agent. Once running, the agent plays the game automatically and saves checkpoints so a learning session can be paused and resumed. This is a research prototype from a team at NVIDIA and is not an official product. The code is released under the MIT license.