VOYAGER: An Open-Ended Embodied Agent with Large Language Models
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi "Jim" Fan, Anima Anandkumar NVIDIA • Caltech • UT Austin • Stanford • UW Madison (2023)
Why This Matters
Traditional artificial intelligence has struggled in open-ended, procedurally generated worlds. Classical Reinforcement Learning (RL) and imitation learning struggle with systematic exploration, interpretability, and generalization because they operate directly on low-level motor commands (like pressing "W" or moving a mouse).
Minecraft is a perfect sandbox for testing lifelong learning. It imposes no fixed path or storyline, requiring players to explore vast 3D terrains, gather resources, craft tools, and unlock a complex tech tree. For an agent to succeed here, it must act like a human: propose its own tasks, write code to execute them, adapt when things break, and remember what it has learned.
The Big Idea
Instead of outputting direct mouse/keyboard controls, VOYAGER uses executable JavaScript code as its action space. By interacting with GPT-4 via blackbox queries, VOYAGER generates code that calls high-level control APIs (via Mineflayer).
This paradigm shift means VOYAGER doesn't need model parameter fine-tuning. Instead, it builds a Skill Library of reusable, compositional code blocks. Because these skills are modular programs, they can be stacked, edited, and transferred to completely new worlds without catastrophic forgetting.
How It Works
The Automatic Curriculum
To ensure open-ended exploration, VOYAGER uses an automatic curriculum driven by GPT-4. It queries the model to propose tasks that are "challenging but manageable" based on the agent's current inventory, nearby blocks, and completed tasks. It operates on a bottom-up novelty search, aiming to "discover as many diverse things as possible."
Interactive Curriculum Simulator
Select a scenario to see how GPT-4's curriculum reasons and proposes the next task based on the agent's state.
Since you have a wooden pickaxe and some stones, it would be beneficial to upgrade your pickaxe to a stone pickaxe for better efficiency.
The Skill Library
The Skill Library is VOYAGER's core memory. Each skill is stored as executable JavaScript code. When faced with a new task, VOYAGER queries the library using vector embeddings of the task description to retrieve the top-5 most relevant skills. This allows the agent to build complex behaviors compositionally.
Interactive Skill Library Browser
Click on a skill to view its description, retrieved relevance, and actual Mineflayer JavaScript code generated by GPT-4.
async function craftWoodenPlanks(bot) {
const logNames = ["oak_log", "birch_log", "spruce_log"];
const plankNames = ["oak_planks", "birch_planks", "spruce_planks"];
const logInInventory = logNames.find(logName => bot.inventory.count(mcData.itemsByName[logName].id) > 0);
if (!logInInventory) {
bot.chat("No wooden log in inventory. Mining a wooden log...");
await mineWoodLog(bot);
}
const logIndex = logNames.indexOf(logInInventory);
const plankName = plankNames[logIndex];
bot.chat(`Crafting 4 ${plankName}...`);
await craftItem(bot, plankName, 1);
}Retrieval description: "First check if a wooden pickaxe is in the inventory. If not, craft one. If available, equip and mine..."
Iterative Prompting Mechanism
Since LLMs rarely write bug-free code on the first attempt, VOYAGER uses an Iterative Prompting Mechanism. It feeds three types of feedback back to GPT-4 to refine the program:
Intermediate execution states (e.g., "I cannot make stick because I need: 2 more planks").
Compiler or interpreter errors from the JS engine (e.g., "No item named acacia_axe").
A secondary critic LLM inspects the final state to verify if the proposed task was actually completed.
Iterative Prompting Loop
See how VOYAGER self-corrects code using compiler errors and environment state checks.
Step 1: Initial Prompt Construction
1 of 4Experimental Results
VOYAGER was evaluated against state-of-the-art LLM agents including ReAct, Reflexion, and AutoGPT. The metrics prove the immense advantage of code-as-action spaces coupled with lifelong learning libraries.
Metric Comparison
Limitations & Open Questions
High API Cost
GPT-4 queries are computationally expensive and incur significant monetary costs (15× more expensive than GPT-3.5). However, GPT-3.5's code generation performance drops off significantly (5.7× fewer items).
Lack of Vision
VOYAGER operates purely on text API feedback. It does not "see" the 3D world, which occasionally leads to failures in spatial construction tasks unless guided by human visual feedback.
Hallucinations
The curriculum occasionally proposes non-existent items (e.g., "copper sword" or "copper chestplate") or attempts to use invalid fuel like cobblestone in furnaces.
Self-Verification Flaws
The critic agent occasionally fails to recognize success signals, such as not identifying "spider string" in the inventory as proof of defeating a spider.
Glossary
@article{wang2023voyager,
title={Voyager: An Open-Ended Embodied Agent with Large Language Models},
author={Wang, Guanzhi and Xie, Yuqi and Jiang, Yunfan and Mandlekar, Ajay and Xiao, Chaowei and Zhu, Yuke and Fan, Linxi and Anandkumar, Anima},
journal={arXiv preprint arXiv:2305.16291},
year={2023}
}