Season 15 division 4a results

Final crosstable



The number of participating engines continues to grow, this season there are 20 candidate engines in division 4, in season 14 there were 12. As a result division 4 is divided into two, 4a and 4b, each group plays a single DRR. The top two engines from each group will play a 2 DRR playoff to decide the two engines that will promote to division 3. The top 4 engines from each group will participate in the cup event later in the season.

The two qualifiers from division 4a are Nemorino and RubiChess. Nemorino was relegated from division 3 and was the clear winner of this group with 13/18 wins and one loss. RubiChess is new to TCEC, it qualified through a tiebreak with Winter due to 2 head-to-head wins. Winter and Rodent will play in the cup event. There were 4 more engines in this group that are new to TCEC: Cheese, Minic, Topple and Igel. These engines ranked last in this group. It should be noted that there were no crashes, unusual considering there were so many new engines. Well done to the authors and administrators.

The TCEC website continues to evolve and improve. The new archive site has the same look as the live games site, you can see the new design in the table above. There is a new administrator team this season, making sure the games run smoothly, fixing and updating the websites and keeping the public informed through the chat. Well done and keep up the good work!

Season 15 division 4a statistics

A summary statistics table of previous stages and seasons. 

Draw rate, wins

 
Final draw rate was 21.1%. This is very low for TCEC, the lowest draw rate in a stage at least since season 7.


Game termination

The three most common game termination causes were:
65.6% - TCEC win rule
18.9% - SyzygyTB
8.9% - TCEC draw rule

There were no crashes in the stage, quite unusual for a stage with several engines new to TCEC. Well done to the authors and the organizers!

Moves per game


Median= 59.8
Average= 64.9

There were 8 games longer than 100 moves, the longest was 135 moves (Topple - Minic, game 54, Topple won).

Time per game (hours)


Median= 1:03
Average= 1:03

Openings

There were 2-move book openings in this stage chosen by Cato. The first letter of the ECO codes was distributed as follows:


The engines had a lot of freedom to choose the opening variant. Yet 64.4% of the game pairs repeated the same ECO code twice, and 62.2% repeated the same opening variant. In 15.6% of the game pairs the ECO first letter was not repeated twice.

Reverse pairs, wins


Reverse pairs, same moves 

Pairs of reverse games diverged very quickly, 46.7% diverged immediately out of book, 77.8% of the pairs diverged at most after 1 move. The longest repeated sequence of moves was 11 plys (Rodent - Cheese, games 4 and 49, Sicilian Alapin variation, two wins for white)

S14 bonus match, Leela - Stockfish, bookless openings

Leela beat Stockfish in a 100 rapid game match with no book, the end result was 16-4 with 80 draws. Leela learned a lot about openings in its training, and it appears to be better than Stockfish from the initial position, at least in shorter time controls. The match also shows that when the engines are allowed to choose the openings we get far less variety than what we usually see in TCEC.

In the 50 games Leela was white it always started with e4, Stockfish played either c5 (Sicilian) or e6 (French). Most of the Sicilians were a Najdorf, English attack variation, 30 games (60%) started with the same 15-ply opening. The French Defense games were mostly Steinitz Boleslavsky variations, 16 games (32%) had the same 13-ply opening. After 20 plys there were only 12 different sequences with information content ( = distribution entropy) equivalent to a space of size 6.4.

Stockfish played e4 in 76% of its white games, it started 24% with d4. Leela answered e4 with e5, leading to many Ruy-Lopez games and a few Italian games. 25 games (50%) started with an 16-ply Berlin Defense line, 11 games (22%) were Giuoco Pianissimo and more variable. Leela answered d4 with Nf6, and this led to QGD games, 7 games (14%) started with a 16-ply sequence in the Vienna variation. After 16 plys there were 19 different sequences with information content ( = distribution entropy) equivalent to a space of size 7.9.

Leela is more stubborn than Stockfish in its opening moves. All the major opening classes in the first few moves - Sicilian, French, Ruy Lopez, Italian, QGD - were determined by Stockfish either in black or in white. Stockfish expanded the opening tree almost every move in at least one opening line, again in both white and black. Leela, on the other hand, almost always played the same move when presented with the same position. In the first 24 plys of 100 games I found only 3 cases where Leela expanded the opening tree, on move 7 as black in QGD (game 29), on move 10 as black in Ruy Lopez (games 45 vs 97), and on move 10 as white in French (game 84).

As white Leela won 6/30 in the Sicilian Najdorf, and lost 2/30. In the French Defense Leela won 5/17 games and lost none. As black in the Italian games Leela won 4/11 and lost 1/11, not a good choice for Stockfish. In the Ruy Lopez Leela won 1/27, the rest were draws. Stockfish won 1/8 in the QGD.

Season 14 superfinal statistics

A summary statistics table of previous stages and seasons. 

Draw rate, wins


Final draw rate was 81%.


Game termination

The three most common game termination causes were:

37% - SyzygyTB
29% - TCEC draw rule

19% - TCEC win rule

Moves per game


Median= 83.5
Average= 99.5

The games were much longer than in previous TCEC stages. This was mainly because Leela's evaluation is designed differently, it is reluctant to lower its eval in drawn positions and the draw rule application is delayed. There were 40 games longer than 100 moves, the longest was 362 moves (game 86, draw, new record for longest game in TCEC history).

Time per game (hours)


Median= 4:35
Average= 4:28

This is also longer than usual, not by a lot because of the increment of 15 seconds.

Openings

There were variable length book openings in this stage chosen by Jeroen. The first letter of the ECO codes was distributed as follows:


The book determined the opening in all cases, all the game pairs repeated the same ECO code twice and repeated the same opening variant.

Reverse pairs, wins


Reverse pairs, same moves 

Pairs of reverse games diverged very quickly, 70% of the pairs diverged at most after 1 move. The longest repeated sequence of moves was 19 plys (games 43-44, Sicilian Velimirovic attack, two draws).     

Season 14 superfinal, games 91-100

Stockfish won the season 14 superfinal. The final score is 10-9 with 81 draws.
The last 10 games ended in a draw and weren't that exciting, except for the anticipation at the beginning of each game to see whether it would turn out to be decisive. In the end Stockfish held on to the small one point lead to win the match.

Superfinal summary
This superfinal match has been one of the best I've seen in TCEC, if not the best so far. After several seasons of Stockfish dominance this superfinal was really competitive and the result was very close until the very end. Leela's journey in this season was remarkable, it got better every time it was updated on its way up from division 3. I was surprised that it was so close to winning based on its performance in the premier division. Recall that it was very close to Houdini and Komodo, Stockfish had a much better score.

There were many decisive games at the start of the match, Stockfish led 6-3 after 22 games, including one game pair where Stockfish won from both sides. Leela closed the gap to 6-6 after 30 games, and after 19 straight draws Leela took the lead, after 60 games it was ahead 8-7. Leela continued to lead by 1-2 points but Stockfish kept close until it equalized to 9-9 after 80 games. Then Stockfish won a game to lead 10-9 and held on to this small lead until the end.

Stockfish may have won this superfinal, but it seems likely that Leela (NN based engines in general) may be the future of computer chess. There were many games where Leela played the opening much better than Stockfish, getting to positions where Stockfish realized it was worse or losing. Leela's closing skills are still lacking, it allowed Stockfish to escape many times. There was even one game where Leela failed to convert a 7-man tablebase win, despite having access to 6-man tablebases. Some of Leela's losses were a result of not looking far enough ahead and making a losing move, sometimes realizing it was losing immediately and sometimes many moves later. To summarize, it seems Leela is already better than Stockfish in some aspects of the game, and it needs to improve its skill in converting wins and calculating ahead. There are suggestions of a hybrid approach, who knows how much better engines can get.

=======

Games 91-92 started with a 23 ply book sequence in the Frence Defense, Winawer variant, where both kings are still in the center and white has a king side passer. In game 91 Stockfish castled long and Leela kept its king in the center. Stockfish gave 3 pawns and moved pieces forward on the king side, Leela's eval climbed over 1.5. Stockfish then gave a rook for a bishop, its two knights attacked the king side and Leela had to be careful. The tension was resolved in a series of exchanges, Stockfish ended the game in a perpetual check. In game 92 Stockfish went a pawn up and pushed its passer to the 7th rank. Evals stayed below 1 and slowly came down. Leela blocked the pawn and slowly exchanged pieces, by move 47 the passer was gone and the game reached a rook ending. The game was adjudicated 20 moves later.

Games 93-94 started with black a pawn up and white with much better developement and a king that cannot castle. In game 93 Leela's eval started over 1. Stockfish gave back the pawn and created a passer on the queen side, then pushed it all the way to the 2nd rank. Leela controlled the center and the black king remained uncastled. Leela temporarily sacrificed a knight and opened the king side. Leela's eval jumped over 2 and then after a series of exchanges only RRN vs RRN remained on move 39. Leela tried to give a rook for a knight, but couldn't break Stockfish's defense. The game ended in a tablebase draw on move 122. In game 94 the evals stayed under 1. Stockfish traded NN for R and opened the king side, Leela gave a rook for a bishop that resulted in an unusual imbalance RR vs BBN. Evals dropped to 0 and the game was adjudicated by the draw rule on move 36, with perpetual check expected in the PV.

Games 95-96 were another King's Gambit variant, black started a pawn up. In game 95 Leela gave a second pawn and the queens were off early. Both kings moved and didn't castle, evals were negative but started to come down after move 20. Leela equalized the material, Stockfish still had a king side passer it pushed to the 3rd rank. Stockfish gave material to clear the path for its passer, then won the material back after the passer promoted. The game reached a B vs R ending and a tablebase draw. In game 96 there were many early exchanges and only RBN vs RNN were left on move 21. Leela's eval was negative over 1, Stockfish's eval dropped to 0 after the engines reduced to a BN vs R ending. The rest of the game was a long shuffle until Leela lowered its eval, and then a tablebase draw again.

Stockfish's eval stayed low in game 97, Leela's eval was under 1. The black king remained in the center, Leela gave a rook for a bishop but didn't get any initiative as a result. Stockfish gave the material back and the engines reduced to a queen ending on move 36. Stockfish ended the game in a perpetual check more than 50 moves later. In game 98 Leela pushed pawns on the queen side and had a space advantage. Only one pair of pawns was exchanged until move 20 and evals were close to 0. The engines started to exchange pieces and pawns gradually, on move 48 Leela gave a knight for pawns and the game reached a RBN vs RB position. Stockfish stopped the black passers on the 2nd rank and the game ended in a tablebase draw.

Games 99-100 started with a 26-ply book sequence in the Sicilian Najdorf opening. In game 99 Stockfish kept its king in the center, its eval was 0 after 2 moves and stayed there throughout the game. It opened the queen side and attacked with its queen, Leela's king was forced to run to the center. Leela exchanged queens so its king would be safe despite being exposed, on move 34 only RBN vs RBN remained. After exchanging rooks Leela went up two pawns, but the bishops were of opposite colors and  there wasn't enough material for a win. The game ended in a tablebase draw. Stockfish was almost sure of winning the match at this stage, one point ahead and a white bias in the last game. Leela castled short in game 100, then pushed pawns on the queen side towards the white king. Stockfish allowed a black pawn to get to b2 and hid its king behind it. Leela concentrated its major pieces on the queen side and the white king walked to the center. The engines reached a standoff, evals came down and the engines started shuffling. Thankfully, Leela reduced its eval close to 0 and the game ended quickly by the draw rule. This draw confirms Stockfish is the winner of the superfinal.

Season 14 superfinal, games 81-90

After 90 games Stockfish leads 10-9 with 71 draws.
Stockfish won a game in black, starting with a very unusual imbalance after book. Leela had another missed win, it had quite a few of these in the match. Stockfish is back in the lead after being behind from game 49. There are only 10 games left, 5 openings, are we going to see drama in the final games?

In games 81-82 black started a pawn up. In game 81 Leela pushed pawns to a6 and h6, two fawn (thorn) pawns! After several exchanges Stockfish captured the pawn on a6, it was two pawns up but its eval was 0 and Leela's eval was over 1. Leela gave a third pawn and reduced to a QRB vs QRB position, its compensation was total control of the dark squares and a mate threat on the long diagonal. Leela gave the rook for a bishop and pawns and the h6 pawn became a passer. However, Leela couldn't push its passer forward and the engines started to shuffle and trade pawns. Leela wouldn't lower its eval and pawn moves kept resetting the 50-move counter. The game ended in a tablebase draw on move 270. In the reverse game 82 evals stayed low throughout the game. Leela gave the pawn back and after a long series of exchanges only RR vs RNN were left on move 27. All pawns were on the king side with Stockfish a pawn up. The rest of the game was shuffling, a few pawn moves and exchanging a pair of rooks. The game was adjudicated on move 150.

Both engines castled long in game 83. Leela pushed pawns on the king side and had a space advantage. Stockfish countered by attacking the queen side and forcing the white king to go to the center. The engines reduced to a QB vs QB position on move 49 and started to shuffle. The game reached an opposite color bishop ending and was adjudicated. In game 84 Stockfish castled short and pushed its king side pawns while Leela kept its king in the center. Leela gave two pawns and reduced to a RBN vs RBB position on move 26, evals were over 1. After the engines exchanged the rooks Stockfish's eval dropped to 0. Despite being two pawns down Leela managed to hold, first with its bishop pair and then in an opposite color bishop ending.

Games 85-86 started with a 24-ply book sequence in the King's Indian Saemisch opening, with an unusual imbalance of Q vs BBPP, and the white king unable to castle. In game 85 evals turned negative after a few moves, Stockfish captured a pawn on the queen side. On move 24 Stockfish's eval jumped over 1.5, it captured another pawn and opened a file on the king side. The black rooks looked dangerous with the white king exposed.

Evals came down after a few moves, it seemed Stockfish didn't have the strength to attack the white king. The engines traded NN for R and made the imbalance even more unusual. Leela was left with only its major pieces, it concentrated them on the king side both as protectors for its king and as attackers on the open files. The engines exchanges a pair of rooks and evals climbed back above 1.5.

Stockfish's eval jumped over 3 and stayed there for a while. Both kings were vulnerable, Leela was down to its last pawn. On move 54 Leela thought it found a way out, it gave its rook for a knight and its pawn became a passer supported by the queen. Leela's eval dropped, it was sure the threat of promotion was enough to force a draw. Stockfish thought it was winning and its eval shot over 7. This went on for a few moves, on move 59 Stockfish saw the mate.

Only now Leela realized it was free to promote its pawn, but its king was in a mating net and there was no defense without losing material. Either way Stockfish was winning and the game ended a few moves later with mate in sight. What a game!

In game 86 Stockfish attacked the king side immediately after book and evals were positive. Leela was careful not to open the king side and it took out the two white knights, one of them for a rook. Stockfish was left with major pieces and after stabilizing the queen side pawns the black king left the king side and the engines started to shuffle. After two sets of almost 50 moves Stockfish's eval dropped to 0, but Leela's eval turned negative. Two more sets went by and then Stockfish's eval jumped again. The game went on and on, in the end breaking the record of longest TCEC game 362 moves - one more than previous record! Almost 7 hours, still a draw. Stockfish wins the game pair and is leading once again.

Queens were off early in game 87. Stockfish's eval was low and became 0 despite Leela having a bishop pair advantage. The files were closed except for an open file on the queen side, Stockfish guarded it with its pieces. On move 31 Stockfish unexpectedly moved a knight and let a white rook advance to b7, Leela's eval immediately jumped close to 3. Stockfish captured the rook for a knight, and realized it was in danger.

The engines played a line they both saw in the PV, Stockfish gave back a rook for a bishop and Leela created an advanced passer in a RB vs RN position. The white king moved forward to support the passer and the black rook moved behind the white pawns. Stockfish created its own queen side passer, forcing Leela to trade passers. Leela was still a pawn up but Stockfish reduced to a rook ending and Leela's advantage was gone. Evals slowly came down, the game was adjudicated on move 114 in a tablebase position. Leela was close to winning this game. 
In the reverse game 88 most pieces remained on the board. Stockfish had a space advantage and evals climbed over 1.5. Leela's pawns stopped moving on move 25 and the engines started to shuffle. On move 42 Stockfish traded its last knight, and without knights it was difficult to penetrate the black fortress. Stockfish agreed to lower its eval only 130 moves later, the game was adjudicated on move 189.

In game 89 all pieces remained on the board for the first 14 moves and the first pawn exchange was on move 23. Leela's eval started over 1 and climbed to around 2 though material was equal and there was no visible advantage. The engines cleared most of the queen side pawns and started to shuffle on move 32. Leela acknowledged that it had nothing 40 moves later and its eval dropped. Leela gave a rook for a bishop and broke through the black pawn line, Stockfish sacrificed a rook and ended the game in perpetual check. In game 90 there were a few early piece exchanges and Leela captured the b2 pawn with its queen. Evals were under 1, both kings did not castle. The engines exchanged queens and opened the queen side, then reduced to a RRN vs RRB position on move 28. After a long shuffle the engines lowered their evals and the game was a 50-move draw.

Season 14 superfinal, games 71-80

After 80 games Stockfish evens the score to 9-9 with 62 draws.
There were 9 draws, in which Leela had 2-4 missed opportunities to win. Then came game 80, a spectacular game. Leela's evaluation was misleading and Stockfish won with two white queens on the board.

There were almost no exchanges in the first 35 moves of game 71. The center and queen side were blocked by pawn lines, Leela concentrated forces on the king side and its eval steadily increased. Stockfish's king ran to the queen side as Leela opened the g file and white pieces moved forward into black territory. Leela's eval peaked over 5, Stockfish also thought it was in trouble with an eval over 2.

Leela's attack seemed to lose steam at this point. The white pieces couldn't attack from the king side, the white king was exposed and Stockfish threatened to attack with its queen. Leela gave a rook for a bishop and got a passer to the 7th rank, Stockfish blocked with a rook and its eval dropped to 0. Leela's attack was over and the game was adjudicated 20 moves later. Was this another missed chance for Leela?

In the reverse game 72 there were more exchanges and the position was open. Evals started around 1 and gradually came down, despite the fact that Leela gave a rook for a knight. Leela had two forward knights near the white king that looked intimidating. In a series of exchanges the game reached a Q vs QN position, Leela 3 pawns up. Stockfish exchanged queens and was in time to stop the white pawns with its knight, resulting in a draw.

There were many early exchanges in game 73, only QBN vs QBB were left on move 26. Leela had a queen side passer, Stockfish's eval was 0. The engines reduced to a queen ending and Stockfish gave checks for almost 50 moves. The game was adjudicated when Leela lowered its eval enough for the draw rule. In game 74 Stockfish was up a pawn with a queen side passer and eval above 1. Leela created a pawn line across the board and the engines started to shuffle from move 22. After about 50 moves there were several exchanges and evals came down, then the engines resumed shuffling. The game ended in a draw by the 50-move rule.

In game 75 evals didn't move a lot after the opening. By move 25 the engines cleared two files on the queen side, all pieces were still on the board. The center was locked, then all rooks and queens were exchanged off by move 35. Evals started to increase slowly though material was equal, Leela's eval reached over 2 and stayed there for another 20 moves. Then Leela pushed pawns on the king side and its eval jumped over 4.5.

Leela had a bishop pair and a queen side passer, Stockfish's pieces had very little space. However, Leela seemed to have no plan how to continue and the engines shuffled for 30 moves, Leela's eval slowly decreasing. The engines cleared the king side pawns and Stockfish's eval dropped. The black bishop was free to move, Leela's advantage was over. The game continued until reaching a tablebase draw on move 159.

There were more early exchanges in game 76, and several minor pieces were off. Stockfish attacked on the king side but wasn't strong enough. Leela gave a pawn in the center and evals came down. Mostly major pieces remained, Leela covered the back ranks and didn't allow the white pieces to get there. The position stabilized and the game was adjudicated by the draw rule.

Games 77-78 started with a 23-ply book sequence in the Benoni Defense. There were many early exchanges in game 77, only QRR vs QRN remained on move 21. Stockfish was two pawns up and its eval was 0. The engines shuffled for a while when Stockfish's eval jumped over 2, it was convinced Leela had an advantage. This lasted for a few moves, then Leela exchanged a pair of rooks and opened the king side, Stockfish's eval fell back to 0. Leela's king was exposed and Stockfish had a perpetual check draw. Leela agreed to the draw on move 117. In game 78 there were only a few exchanges. Stockfish's eval stayed around 1, Leela blocked the board with a pawn line and the engines started to shuffle on move 33. Leela had a central knight that attacked two weak white pawns, forcing Stockfish to keep a defending piece for both. After 30 moves Stockfish opened a file on the king side, and for a brief moment its eval jumped close to 2. Leela remained calm despite being low on time, it captured a pawn on the queen side and evals fell immediately down to 0. Stockfish couldn't attack when it had to keep an eye on the black queen side passers. The engines shuffled and exchanged pieces until adjudication on move 149.

The position was closed in game 79 with very few exchanges. Leela had a space advantage, its eval was over 1 and it increased slowly. The queen side was blocked, Leela had a knight pair which is better in closed positions, Leela's eval went over 2.5. The engines shuffled pieces and pushed pawns on the king side until it was also locked. On move 90 Stockfish gave a rook for a knight and the engines continued to shuffle with evals coming down. On move 140 Leela traded its other knight and gave up its advantage. The game was adjudicated on move 191.

In game 80 the position was again closed. Stockfish had a space advantage and eval over 1, it preferred trading a knight for a bishop so Leela had the knight pair. Stockfish pushed pawns on the king side, Leela blocked with its pawns. Evals started to increase when Stockfish gave a knight for pawns and broke through the center. Stockfish followed with a knight for rook trade and its eval jumped over 3.



After exf7+ Leela moved its king to h7, assuming that Stockfish would take the knight, more or less equalizing material. Stockfish however chose to promote on f8, safe because of the pinned bishop. This cost a white rook but now Stockfish had an unusual 2 queens. Stockfish's eval shot up, Leela was incredibly cool.



Leela's eval started to react to the situation only on move 47, much too late. Stockfish's queens dominated the board and Leela couldn't stop them. The game ended with mate a few moves away. An incredible game, quite unexpected behavior by Leela. Stockfish wins the game pair and evens the score of the match.