Minecraft bot Voyager programs itself with GPT-4

Voyager uses GPT-4 to control a learning Minecraft agent in a pixel world. Instead of reinforcement learning, Voyager relies on code generation.

Researchers from Nvidia, Caltech, UT Austin, Stanford, and ASU present Voyager, the first lifelong learning agent to play Minecraft. Unlike other Minecraft agents that use classic reinforcement learning methods, for example, Voyager uses GPT-4 to continuously improve himself. It does this by writing, improving, and passing code stored in an external skill library.

The result is small programs that help you navigate, open doors, mine resources, make a pickaxe, or fight zombies. "GPT-4 opens up a new paradigm," says researcher Jim Phan of Nvidia, who advised the project. "In this paradigm, 'training' is code execution, and the 'trained model' is the skill base that Voyager iteratively builds."

The Voyager consists of three main components:

1. An iterative hinting mechanism that includes feedback from the game, execution errors, and self-checking for program improvement.
2. Skills library with code to store and retrieve complex behaviors.
3. Automated curriculum for maximum learning.

Agent Voyager Minecraft learns in context

Minecraft Agent learns in an iterative way: Voyager writes a program using GPT-4 to achieve the goal, and uses feedback from the game environment and possible Javascript errors to refine the program using GPT-4. In this way, Voyager gradually builds a library of skills and stores successful programs in a vector database. Complex skills are formed from simpler ones.

To explore Minecraft's diverse world, the team uses an automated learning program that suggests appropriate exploration tasks based on the agent's current skills and the current state of the world. For example, an agent learns to collect sand and cacti in the desert before digging for iron.

All of this together creates an agent who is constantly learning and can perform various tasks. The team conducts all experiments in the MineDojo environment.

Currently, Voyager can only build homes with human feedback.

The team compares Voyager to other language model-based agents such as ReAct, Reflection, or Auto-GPT in Minecraft. Voyager found 63 different objects in 160 iterations of cues - 3.3 times more than the next best approach, the team says.

Auto-GPT makes Voyager travel a lot: In general, the Minecraft agent travels more than twice the distance and visits more biomes. Auto-GPT and other methods, on the other hand, often get stuck in their local area.

The skill library created by Voyager is also compatible with Auto-GPT: the AI agent in Minecraft achieves significantly better results with it, but still lags behind Voyager.

Voyager currently only works with text and cannot see what is happening in the block world. Therefore, it can't build houses. However, in an early experiment, the team used humans to give the agent visual feedback - so Voyager can learn to build houses and portals to Nether, for example.

More information and examples are available on the project page Voyager.

The code is available at GhitHub.