Saturday, December 24, 2022

Season 23 Noomen Extra bonus, opening tree analysis

Season 23 ran a bonus event called the Noomen Extra bonus, where the top 3 engines (Stockfish, Leela and KomodoDragon) played game pairs against all the engines in the premier division and league 1 - 14 engines including the top3. The event was run in two parts, first only the top3 engines played against themselves, and then the the games between the top3 and the other engines. Four book sequences were used in the top3-top3 games: e4 g6 (Robatsch / Modern), d4 Nf6 c4 g6 Nc3 Bg7 (KID), e4 c5 (Sicilian), e4 c6 (Caro-Kann). In the second part of top3-non top3 games the last book sequence was replaced by d4 Nf6 c4 c5 d5 e6  (Benoni defense).

Each top3 engine played 13 game pairs in each of the 4 book sequences (except the Benoni, only 11 game pairs each). I used this dataset to get a current snapshot of how the top3 engines play openings. In particular I was interested in how fast the opening tree expands, or in another words: to what extent does a top3 engine play the same move when given the same position? In previous analyses I did in season 14 and season 18 Leela was very rigid, in almost all cases it played the same move in the same position. Stockfish was more willing to try several options, expanding the opening tree. Looking at the game results it appears that Leela's opening play was better than Stockfish's then, it usually had an advantage after the opening and in some cases won the game. These events took place before Stockfish used NNUE, since then its opening play has improved and it is rarely outplayed in the opening.

Several observations after doing this analysis:

1. All top3 engines are flexible in their opening moves, there were many occasions where they chose to play different moves in identical positions. They can have different preferences, where one engine always plays one move in a position and another engine plays a different move or chooses between several options in the same position.  

2. I can't say which top3 engine is more or less flexible in the opening, nor can I say which engine is better in the opening. All top3-top3 games ended in a draw.

3. The top3 engines beat the rest of the field easily, it was almost embarassing in some of the openings. The opening bias was much lower than what we usually see in the premier division, not to mention the superfinal.

4. KomodoDragon won the event, consistent with its performance in the premier division of the last few seasons. In long matches against Leela or Stockfish it lost badly but it has an advantage against lower ranked engines.

Robatsch (Modern): e4 g6

The first ply after book in all 72 games was d4. In the second ply all top3 engines played both Bg7 and d6. Stockfish preferred Bg7, Leela preferred d6, KomodoDragon had no preference. These were also the choices of the other engines.

There were more expansions in the 3rd ply. After d4 Bg7 the top3 played Nc3 (dominant choice for KomodoDragon and Stockfish), c4 or Nf3. After d4 d6 the engines played Be3, Nc3 or c4.

From ply 4 the opening trees continued to expand. In total KomodoDragon was responsible for 10 expansions, Leela for 9 and Stockfish for 8 (the full number of expansions until all games are unique is 24 for each engine, the set of opponents are responsible for the remaining expansions). After 20 plys all 72 games had unique move sequences. The longest repeat was 19 plys long between game 16 (Leela beat ScorpioNN) and game 62 (KomodoDragon beat rofChade).

The above analysis ignores transpositions, which were quite frequent in this opening. For example, after 10 plys there were 62 unique move sequences, but only 44 when taking transpositions into consideration. The two most frequent sequences were:
d4 Bg7 Nc3 d6 Be3 Nf6 Qd2 c6 Nf3 O-O - 7 times including transpositions, all top3 black draws against non-top3 engines. The longest repeat for this line was 28 plys in games 31 (RubiChess - KomodoDragon) and 49 (SlowChess - KomodoDragon).
d4 Bg7 Nc3 d6 Be3 Nf6 Qd2 c6 Bh6 Bxh6 - 5 times including transpositions, all top3 black draws against non-top3 engines. The longest repeat for this line was 58 plys (!!), games 23 Ethereal - Stockfish) and 65 (rofChade - Stockfish).

There were 20 decisive games in this opening, all wins by top3 engines against non-top3 engines. There was one black win, probably not related to the choice of openings moves. This may be true also for white wins, but since there are more of these we can give some numbers:
d4 Bg7 Nc3: c6 was followed by h3 (3/3 wins), other choices were better (d6 or d5, 5/18 wins).
d4 d6 Be3 Nf6 Nc3: a6 was followed by h3 (3/3 wins), Bg7 was better (0/2 wins).
d4 Bg7 Nc3 d6 Be3: c6 was followed by f4 (2/2 wins), other choices were better (a6 or Nf6, 7/33 wins)

The most frequent move sequence of length 6 was d4 Bg7 Nc3 d6 Be3 a6, 24 games with transpositions. Playing h4 here led to 5/6 wins, other moves (Qd2,a4,Nf3,f3) only 2/18 wins. All the top3 engines played h4 in less than half their games that started with this sequence. Interestingly, this move sequence appeared twice in top3-top3 games (game 6 Stockfish - KomodoDragon and game 13 Leela - KomodoDragon), In both cases white did not play h4.

KID: d4 Nf6 c4 g6 Nc3 Bg7

The first ply after book in all 72 games was e4. In the second ply the moves played were d6 and O-O, Stockfish and KomodoDragon always played d6, Leela preferred O-O though it also played d6 in a few games. The opening tree expanded on the 3rd ply. The responses to e4 d6 were Be2 (dominant for Stockfish), h3 (dominant for KomodoDragon) and Nf3. After e4 O-O, one of Nf3, Be3, Be2 and h3 was played.

In total KomodoDragon was responsible for 10 expansions, Leela for 10 and Stockfish for only 5. After 45 plys all 72 games had unique move sequences. The longest repeat was 44 plys long between game 89 (Ethereal - Stockfish, draw) and game 113 (Berserk - Stockfish, draw). Including transpositions these games repeated 50 plys.

There were many transpositions in this opening. For example after 10 plys there were 47 unique move sequences, including transpositions there were only 32. The most frequent 10 ply sequence, including transpositions was e4 d6 Nf3 O-O Be2 e5 O-O Na6 Be3 Qe7, it was played 8 times using 6 different move sequences.

There were 15 decisive games in this opening, all wins by top3 engines against non-top3 engines. There was one black win, probably not related to the choice of openings moves. 

The main 4-ply sequences had drawish results:
e4 d6 Be2 O-O - 23 games (including transpositions), 4/23 wins
e4 d6 Nf3 O-O - 19 games (including transpositions), 1/19 wins

In 20 games the 3rd ply after book was h3, leading to the sequence e4 d6 h3 O-O in 10 of these games, including transpositions. Be3 was then played in 9 games, with 5/9 wins. Breaking down this sequence by the engine in white: KomodoDragon (4/6 wins), Stockfish (1/1 wins), Leela (0/1 wins), other (0/1 wins).

Sicilian: e4 c5

This opening was more drawish (only 5 white wins) and diverged more slowly.

The first ply after book was Nf3 in 71/72 games, Nc3 was played once. After Nf3, 64 games continued d6, the other options were Nc6 (3) and e6 (2). The 3 games starting with Nf3 Nc6 continued Bb5 e6, the Rossolimo attack variant, with 2/3 wins for white. Most of the Nf3 d6 games became Najdorf Sicilians, Nf3 d6 d4 cxd4 Nxd4 Nf6 Nc3 a6 (57/64 with transpositions). The most frequent continuations were f3 e5 Nb3 (21), Be3 e5 Nb3 (15). After 5.. a3 Leela always played f3, KomodoDragon and Stockfish tried 3-4 different moves.

In total KomodoDragon was responsible for 4 expansions, Leela for 5 and Stockfish for 7. After 38 plys all 72 games had unique move sequences. The longest repeat was 37 plys long between games 1 and 11 in the top3-top3 part, Leela - KomodoDragon and Leela - Stockfish, both draws. 

There were transpositions in this opening as well. For example after 16 plys there were 52 unique move sequences, including transpositions there were only 38. The longest repeat with transpositions was 62 plys, in fact games 145 (ScorpioNN - KomodoDragon) and 149 (ScorpioNN - Stockfish) were identical !!

Benoni defense: d4 Nf6 c4 c5 d5 e6

This was the most unbalanced opening tested, with 27 wins for top3 engines as white against non-top3 engines. This opening was not used in the top3-top3 games, therefore there are only 66 games.

The first ply after book was Nc3 in 64/66 games, Nf3 was used in 2 games. There were two main lines played:
Nc3 d6 e4 g6 - 40 games including transpositions, 6/40 wins. Top3 engines almost always chose this line as black and these games ended in draws. Of the non-top3 engines, Ethereal Stoofvlees and Igel preferred this line as black. Still they lost most of these games.
Nc3 exd5 cxd5 d6 e4 g6 f4 - 23 games, 20/23 wins. Only non-top3 engines were black in these games, almost always losing.

In total KomodoDragon was responsible for 4 expansions, Leela for 4 and Stockfish for 3. After 48 plys all 66 games had unique move sequences. The longest repeat was 47 plys long between game 205 (Seer - KomodoDragon) and 233 (RubiChess - Stockfish), both draws. 

Taking transpositions into account divergence was slow. For example after 20 plys there were 49 unique move sequences, including transpositions there were only 29. The longest repeat with transpositions was 55 plys, games 228 and 240 (Stockfish - Revenge/Koivisto, two wins).