AI Go Championship Kicks Off: Large Models Show Amateur-Level Play, While AlphaZero Remains the Ultimate Benchmark

The first-ever “AI Chess Grandmaster Championship” has officially kicked off, with the tournament hosted by Google and Kaggle launching a three-day live-streamed event starting August 5. The first day’s results are now in, revealing a clear dominance of top-tier models over their competitors. In the upper bracket, two Chinese large language models—DeepSeek-R1 and Kimi K2 Instruct—both suffered decisive 0:4 losses, falling to o4 mini and o3 respectively. In the lower bracket, Google’s Gemini 2.5 Pro defeated Claude Opus 4, but its smaller sibling, Gemini 2.5 Flash, was bested by Grok 4. All matches followed a clean sweep format, with winners claiming all four games. However, game durations varied significantly. The fastest match was o3 versus Kimi K2, lasting under 30 minutes—largely due to Kimi K2’s repeated illegal moves. The model repeatedly attempted to move the queen from d1 to d4, a violation of basic chess rules. In contrast, the o4 mini vs. DeepSeek R1 match lasted nearly two hours, indicating a much closer and more balanced contest, with both models making complex decisions over time. Tomorrow’s matches will feature a high-stakes showdown between o4 mini and o3—the same model family—while Gemini 2.5 Pro faces off against Grok 4. These live events are considered exhibition matches, but Kaggle plans to run extensive behind-the-scenes matchups to generate a statistically robust “AI Chess Grandmaster” leaderboard. The competition takes place on Kaggle Game Arena, a new benchmark platform developed jointly by Kaggle and Google DeepMind. Unlike traditional static AI evaluations, Game Arena uses real-time, adversarial gameplay to assess AI performance. Models compete in multi-round matches under clear win-or-lose conditions, with results serving as direct measures of capability. This dynamic testing approach helps avoid the “memorization” or “cheating” issues seen in static benchmarks, offering a more authentic view of how AI systems perform under pressure. The platform’s focus on games like chess is intentional—these games provide structured, rule-based environments with unambiguous victory conditions, enabling rigorous evaluation of strategic reasoning, long-term planning, and adaptability. The foundation for such testing lies in the success of AlphaZero, Google DeepMind’s 2017 breakthrough. Using only self-play and reinforcement learning, AlphaZero mastered chess in just hours, defeating the then-strongest engine, Stockfish, with overwhelming consistency. However, the models in Kaggle’s tournament are not specialized chess engines. Instead, they are general-purpose large language models (LLMs), which currently operate at an amateur level and frequently make fundamental errors—such as illegal moves, absurd resignations, or stubbornly repeating flawed strategies even after being corrected. Despite these shortcomings, the models offer a unique advantage: they can generate detailed “thought processes” for each move. This transparency—showing how the AI reasons through a position—is something traditional chess engines cannot provide. By observing these internal justifications, researchers gain insight into how AI approaches complex decision-making. The tournament uses a single-elimination format with pre-tournament warm-up rounds to seed the bracket. Higher-ranked models face lower-ranked ones to ensure balanced matchups and prevent top contenders from meeting too early. Each game follows standard chess rules, and results are recorded in real time. Kaggle maintains a live, Elo-style ranking system that tracks model performance dynamically. The scoring model uses a Gaussian-based estimation system: winners gain points, losers lose them, and draws bring both scores closer to the average. The update size depends on how much the result deviates from expected outcomes and the uncertainty (σ) in each model’s current rating. As more games are played, σ decreases, and ratings stabilize—mirroring how human chess ratings are updated. All models participate via text-only input and output. They receive the current board state in Forsyth-Edwards Notation (FEN) and the game history in PGN format. They must respond with a legal move in Standard Algebraic Notation (SAN). If a move is invalid, the model gets up to four attempts (one initial plus three retries). If it fails to produce a valid move, it loses the game. Each move has a 60-minute response time limit to maintain pacing. The live stream also captures the model’s internal reasoning before each move, providing valuable data for post-game analysis. Why chess? Because it offers a clear, measurable success signal and demands deep strategic thinking. From opening principles to endgame tactics, models must process dynamic positions, recall past moves, adapt to opponents’ strategies, and even infer intentions—skills that mirror real-world decision-making in business, policy, and complex planning. Currently, most LLMs are not optimized for chess. Unlike engines such as Stockfish or AlphaZero, they lack access to specialized databases or brute-force search capabilities. Google acknowledges this: “Professional engines have maintained superhuman performance for years. Any general-purpose model will struggle against them. Today’s LLMs are not game-specific, so their chess skills remain far behind.” Still, the goal is not immediate dominance. The short-term aim is to help general models improve. Long-term, Kaggle and Google hope to push LLMs toward mastering new games and eventually surpassing current benchmarks—transforming them into more robust, strategic thinkers. For more information, see: https://www.chess.com/article/view/chatgpt-gemini-play-chess https://www.chess.com/news/view/which-ai-model-is-the-best-at-chess-kaggle-game-arena https://blog.google/technology/ai/kaggle-game-arena/ https://www.theregister.com/2025/07/14/atari_chess_vs_gemini/ https://www.kaggle.com/benchmarks/kaggle/chess-text/leaderboard