Stefan Pohl Computer Chess

LC0 / Neural Nets versus Stockfish testing

Playing conditions:

Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM

Speed:  Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off): 9000 kn/s, Lc0 (with old 32930 20x256 net): 16000 n/s in starting position.

Hash / NN Cache: 4096 GB Hash for Stockfish / 5000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: NBSC Advanced Armageddon Noomen 3-moves (250 openings).  Learn more about Advanced Armageddon in the "NBSC Armageddon openings"- section and download the NBSC-Armageddon package right here

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 2'+1'' and Stockfish 3'+1.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 8 minutes, one 500 games-testrun takes around 2.5 days.

Each Lc0 / Neural Net plays 500 games vs. Stockfish with my new NBSC Advanced Armageddon openings. After the testrun is finished, all games are rescored with my armageddonize_advanced-tool. Means: 

Win for white = 1 point for white
Draw = 1 point for black
Win for black = 2 points for black 

Learn more about my new NBSC Advanced Armageddon openings and the advanced scoring system in the "NBSC Armageddon openings"- section.

Latest update: 2020/06/02 Lc0 0.25.1 63651. Next NN-testrun Lc0 0.25.1 t60-3972.

Download all played games (non-armageddonized) here

500 NBSC-Advanced-Armageddon games each testrun (= a win for Black is 2 points for Black and a draw is a 1 point-win for Black). vs. Stockfish 200418 (SPCC-Elo: 3568 (Contempt set to 0) (around +14 Elo stronger than Stockfish 11 (SPCC-Elo: 3554)).

The errorbar of each result is +/- 20 Elo. But mention, that the usage of my NBSC-Armageddon openings spreads the Elo-results around 2.25x wider, than using classical openings for testing(!), so with classical openings, you would need an errorbar of +/- 9 Elo for the same statistical quality of the results (= the rankings of Lc0 nets here). And for an errorbar of +/- 9 elo, you need around 3000 games, not 500, which means 6x more games (and 6x more PC-time)!!

Learn more about that revolution in computerchess in the "NBSC Armageddon openings"- section of my website.

1  Lc0 0.24.1 LS 14.3              : 3644 513 (+311,=  0,-202), 60.6 %
2  Lc0 0.24.1 LS 14.2              : 3633 520 (+308,=  0,-212), 59.2 %
3  Lc0 0.25.1 t40-1541             : 3583 516 (+269,=  0,-247), 52.1 %
4  Lc0 0.25.1 t60-3010             : 3582 514 (+267,=  0,-247), 51.9 %
** Stockfish 200418 ************** : 3568 SPCC-Elo *******************
5  Allie 0.6 LS 14.3               : 3558 519 (+252,=  0,-267), 48.6 %
6  Lc0 0.25.1 42850                : 3556 522 (+252,=  0,-270), 48.3 %
7  Lc0 0.25.1 63651                : 3554 517 (+248,=  0,-269), 48.0 %
8  Fat Fritz 1.1                   : 3530 523 (+233,=  0,-290), 44.6 %
9  Lc0 0.25.1 63305                : 3530 512 (+228,=  0,-284), 44.5 %
10 Lc0 0.25.1 32930                : 3483 515 (+196,=  0,-319), 38.1 %
11 Lc0 0.25.1 11260                : 3408 521 (+149,=  0,-372), 28.6 %

Mention, the number of games is a little bit too high, because the (rare) wins
of Black are doubled in the pgn-file, which is given to ORDO, because of 
Advanced Armageddon Scoring (= a win for Black is 2 points for Black). 
That trick of doubling these games is the only possibility to make
ORDO count a win of Black as 2 points...

Games        : 5692 (finished)

White Wins   : 2791 (49.0 %)
Black Wins   : 2901 (51.0 %)
Draws        : 0 (0.0 %)

Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus Stockfish. Because Lc0 vs. Stockfish is definitly the most interesting head-to-head competition of NN vs. AB-engines. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here