LLMs Create a SELF-IMPROVING 🤯 AI Agent to Play Settlers of Catan

This video dives into the fascinating world of autonomous, self-improving AI agents, specifically focusing on a new research paper where Large Language Models (LLMs) learn to play Settlers of Catan. It highlights how these AI agents, built with "scaffolding" around powerful LLMs, can overcome common challenges like maintaining long-term strategic coherence and continuously improve their performance, marking a significant step towards more capable and adaptable AI.

Here's a breakdown of the key insights and technical details:

• Understanding AI Agents: The term "AI agent" is used, even if it's not perfect, to describe large language models enhanced with additional architecture or "scaffolding." This scaffolding provides tools like the ability to write code, take notes, or access documentation, enabling the LLM to perform complex tasks.
• Proven Approach: This LLM-plus-scaffolding approach is well-established, similar to groundbreaking projects like Google DeepMind's Alpha Evolve, the Darwin Godel machine (a self-improving coding agent), and NVIDIA's Minecraft Voyager (an AI that played Minecraft, wrote its own code, and continuously improved).
• The Settlers of Catan Challenge: Settlers of Catan is an ideal game for testing strategic planning because it's complex, involving strategy, math, negotiation, and crucially, elements of chance (dice rolls) and partial observability (not all information is visible at once). This makes it harder for traditional AI methods than "perfect information" games like chess or Go.
• The Core Problem: Long-Term Coherence: A significant challenge for current AI agents is maintaining coherent long-term strategies. While they can perform well initially, their performance often degrades over extended periods. This research aims to create a scaffolding that allows agents to improve over time instead of getting worse.
• The Catanatron Framework: The research uses an open-source, Python-based framework called Catanatron to simulate and play Settlers of Catan games, allowing for rapid testing and iteration of AI agent strategies.
• Benchmarking Self-Improvement: The goal is to see if LLM-based agents can evolve from simple game players to systems capable of autonomously rewriting their own prompts and underlying code.
• Multi-Agent Architecture – The "Recipe" for Success: The system employs a multi-agent structure, which is a proven design for complex AI agents. This structure includes:
  • Analyzer: Evaluates gameplay, identifies weaknesses, and summarizes areas for improvement.
  • Researcher: Handles specific queries about Catanatron or broader Catan strategies using local file access and web searches (which is a brilliant addition, allowing the AI to learn new human strategies).
  • Strategizer: Suggests high-level gameplay strategies or critiques past choices.
  • Coder: Translates proposed changes into concrete code modifications for the player agent.
  • Player: The actual AI that plays the game and is continuously improved.
• The Importance of State Reminders: A key insight for maintaining long-term coherence is continuously reminding the AI agent of its current game state. Unlike previous AI projects that failed by "losing the plot" over time (like the Vending Bench paper), providing consistent state updates helps the LLM agents stay focused and effective. This is a critical takeaway for anyone building AI agents.
• Self-Evolving Mechanism (Agent Evolver): The system begins with a basic template and evolves its abilities over time. The "Evolver Agent" acts as a central coordinator, reading reports from the Analyzer and Researcher to decide on modifications. The Coder agent then implements these changes, and the Player agent tests them in the game. If the modifications improve performance, they are integrated into the agent's code, creating a cycle of recursive self-improvement.
• Accessible Experimentation: The research was conducted on readily available hardware (MacBook Pro 2019 and MacBook M1 Max), indicating that developing and testing such self-improving agents isn't necessarily out of reach for smaller teams or even individuals.
• Model Performance Varies: The researchers tested GPT-4O, Claude 3.7, and Mistral Large.
  • Claude 3.7 was the standout performer, showing a massive 95% improvement over the base agent. It excelled at developing detailed strategic prompts with clear short-term and long-term plans.
  • GPT-4O showed a respectable 36% improvement.
  • Mistral Large performed the least effectively, even dropping performance in some cases.
• The Quality of the Underlying LLM is Paramount: A crucial takeaway is that the better the foundational LLM, the better the performance of the entire AI system. This aligns with the idea of building systems that naturally benefit from advancements in core AI models, rather than trying to fix existing AI limitations. Future, more advanced models are expected to further boost these self-improvement capabilities.
• Potential for Further Improvement: The experiments only ran for 10 evolutionary steps. The results showed continuous improvement, especially with Claude 3.7, even towards the later steps. This suggests that given more time and resources, these agents could achieve even greater levels of performance.
• Exciting Implications: This research is another strong example of AI agents' ability to recursively self-improve. It's helping to refine the "recipe" for building effective AI agents, offering valuable insights into what works and what doesn't. Observing these advancements through the lens of games makes the progress both tangible and exciting.