Anthropic’s Newest AI Wants to Be a Pokémon Master. Here’s Why That’s a Big Deal

On Monday, Anthropic released Claude 3.7 Sonnet, the company’s most capable AI model yet. It also revealed a new capability with implications for the business world: Claude can now play Pokémon, and it’s pretty good at it, at least for an AI. In a blog post detailing the new Claude model, Anthropic wrote that a small internal team had created an interface that enabled Claude to play Pokémon Red, the original Pokémon game released on the Nintendo Game Boy way back in 1996. So, why teach an AI model to play Pokémon? David Hershey, a member of Anthropic’s technical team, tells Inc. that staffers were inspired by a YouTube video in which an original reinforcement learning model was trained to play Pokémon, so they created a virtual environment in which Claude could attempt to play the game. Eventually, around June 2024, Hershey (a self-proclaimed Pokémon fan) took up the idea as a side project, first using it to test the capabilities of Claude 3.5 Sonnet, the new model at the time. He found that while earlier versions of Claude would immediately get stuck, Claude 3.5 could progress further, successfully catching a Pokémon and leaving the starting area of Pallet Town. For the uninitiated, the goal of Pokémon Red is to catch adorable creatures, train them by battling against non-playable characters, and win badges from powerful enemies called Gym Leaders. Pokemon is, of course, wildly popular, and is considered the highest-grossing media franchise of all time. To keep his coworkers, many of whom are also Pokémon fans, up-to-date on his efforts, Hershey started a dedicated Slack channel in which Anthropic employees could monitor Claude’s Pokémon journey. Every time Claude successfully caught a Pokémon or won a battle, more and more people would join the Slack channel, says Anthropic product research lead Diane Penn. The project developed a cult following within the company. Video games have long been used as a method for gauging an AI model’s ability. In the early days of OpenAI, staffers spent years training models to play online multiplayer game DOTA 2. “Defining success is hard,” says Hershey, “But video games happen to be structured in a way where progress is often measurable and linear.” Other games have been important over the decades as AI, or just thinking machines in general, developed. In 2016, Google DeepMind’s AlphaGo beat one of the world’s highest-ranked players of the ancient game Go, and in 2011, IBM’s Watson system beat Jeopardy champions Ken Jennings and Brad Rutter at their own game. And in 1997, IBM’s Deep Blue beat Garry Kasparov at chess. When Anthropic released an updated version of Claude 3.5 Sonnet in September, the model saw slight improvement, but the real breakthrough came in the form of Claude 3.7 Sonnet, the new model released this week. While the previous model got stuck in Viridian Forest, an early area in the game, 3.7 Sonnet was able to go further, collecting three badges from Pokémon gym leaders. So how is Claude able to improve itself? Hershey says that Claude’s new Pokémon skills are the direct result of a new feature called “extended thinking,” which enables the model to take additional time to “think” through how to solve a problem, instead of immediately generating a response. Hershey says a common complaint he’s heard from Anthopic’s customers is that earlier versions of Claude would make a false assumption and then struggle to reverse course. But because of its improved thought process, the new Claude is able to more effectively pivot and try new strategies, meaning it doesn’t get stuck nearly as often as earlier versions. According to Penn, Anthropic decided to include Hershey’s Pokémon benchmark in Claude 3.7 Sonnet’s announcement because the company is slowly moving away from traditional benchmarks in favor of more “accessible” tests that can be understood by a larger group of people. “We’re at a point where evaluations don’t tell the full story of how much more capable each version of these models are,” Penn says. Penn says the benchmark demonstrates Claude’s ability to intelligently make a plan and adapt with new strategies when it runs into a problem. For companies looking to use AI on complex tasks like conducting high quality research or complex financial analysis, the benchmark is proof that models can improve their performance by using reasoning capabilities. By using Pokémon progression as a benchmark, Anthropic is able to educate an entirely new audience about Claude’s capabilities. After learning that the benchmark would be included in the announcement, Hershey and a small team hustled to quickly create an ongoing livestream on Twitch, in which anyone can watch Claude attempt to catch ‘em all. Some users have even said in the livestream’s chat that they were inspired to subscribe to Anthropic’s $18 per month Claude Pro service. Even with its enhanced capabilities, the model is still far from becoming a Pokémon Master, say Penn and Hershey. As of Thursday afternoon, the model had been stuck in Mt. Moon, an early game area that’s notoriously tricky for kids, for over 27 hours. Viewers of “Claude Plays Pokémon” were especially delighted when Claude named its rival character Waclaud, a reference to Super Mario Bros villains Wario and Waluigi, but that actually wasn’t a decision made by the model. “It was in the system prompt,” admits Hershey. Before the livestream was launched, he says, “we ran an internal poll on what we should name the rival, so Waclaud is just a small easter egg from our internal culture at Anthropic.” Could the project’s popularity result in Pokémon becoming a new standardized benchmark for AI? Hershey isn’t quite sure, but based on the response his project has received online, he wouldn’t be surprised to see other AI labs use video games as benchmarks more often. “It’s just a great way to see progress over a long period of time,” he says, “and we’re definitely not the only people who think that’s important.” BY BEN SHERRY @BENLUCASSHERRY