Stefan Pohl Computer Chess

LC0 / Neural Nets versus Stockfish testing

Playing conditions:

Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM

Speed:  Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off): 9000 kn/s, Lc0 (with old 32930 20x256 net): 16000 n/s in starting position.

Hash / NN Cache: 4096 GB Hash for Stockfish / 5000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: NBSC Advanced Armageddon Noomen 3-moves (250 openings).  Learn more about Advanced Armageddon in the "NBSC Armageddon openings"- section and download the NBSC-Armageddon package right here

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 2'+1'' and Stockfish 3'+1.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 8 minutes, one 500 games-testrun takes around 2.5 days. 

Each Lc0 / Neural Net plays 500 games vs. Stockfish with my new NBSC Advanced Armageddon openings. After the testrun is finished, all games are rescored with my armageddonize_advanced-tool. Means: 

Win for white = 1 point for white
Draw = 1 point for black
Win for black = 2 points for black 

Learn more about my new NBSC Advanced Armageddon openings and the advanced scoring system in the "NBSC Armageddon openings"- section.

Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here

Latest update: 2020/09/26: Lc0 0.26.2 J92-205 (since J92-130, there is a clear regression (J92-160, J92-205))

Download all played games (non-armageddonized) here

500 NBSC-Advanced-Armageddon games each testrun (= a win for Black is 2 points for Black and a draw is a 1 point-win for Black). vs. Stockfish 200418 (SPCC-Elo: 3568 (Contempt set to 0) (around +14 Elo stronger than Stockfish 11 (SPCC-Elo: 3554)).

The errorbar of each result is +/- 20 Elo. But mention, that the usage of my NBSC-Armageddon openings spreads the Elo-results around 2.25x wider, than using classical openings for testing(!), so with classical openings, you would need an errorbar of +/- 9 Elo for the same statistical quality of the results (= the rankings of Lc0 nets here). And for an errorbar of +/- 9 elo, you need around 3000 games, not 500, which means 6x more games (and 6x more PC-time)!!

Learn more about that revolution in computerchess in the "NBSC Armageddon openings"- section of my website.

1  Lc0 0.26.2 J92-130 (30x384)      : 3655 521 (+324,=  0,-197), 62.2 %
2  Lc0 0.24.1 LS 14.3 (20x256)      : 3644 513 (+311,=  0,-202), 60.6 %
3  Lc0 0.25.1 LS 15 (20x256)        : 3643 512 (+310,=  0,-202), 60.5 %
4  Lc0 0.26.2 J92-160 (30x384)      : 3635 511 (+304,=  0,-207), 59.5 %
5  Lc0 0.26.2 T60B.7-105 (24x320)   : 3634 519 (+308,=  0,-211), 59.3 %
6  Lc0 0.24.1 LS 14.2 (20x256)      : 3633 520 (+308,=  0,-212), 59.2 %
7  Lc0 0.25.1 LS 15 Kayra4          : 3624 513 (+297,=  0,-216), 57.9 %
8  Lc0 0.26.1 t60-4619 (30x384)     : 3622 522 (+301,=  0,-221), 57.7 %
9  Lc0 0.26.2 J92-205 (30x384)      : 3618 511 (+292,=  0,-219), 57.1 %
10 Lc0 0.25.1 3972_20k_tcec (30x384): 3617 514 (+293,=  0,-221), 57.0 %
11 Lc0 0.26.2 65100 (24x320)        : 3616 512 (+291,=  0,-221), 56.8 %
12 Lc0 0.26.1 J92-100 (30x384)      : 3609 510 (+285,=  0,-225), 55.9 %
13 Lc0 0.26.1 t60-4585 (30x384)     : 3605 514 (+284,=  0,-230), 55.3 %
14 Lc0 0.25.1 sv-1810 (20x256)      : 3599 514 (+280,=  0,-234), 54.5 %
15 Lc0 0.25.1 t60-4175_mlh (30x384) : 3594 516 (+277,=  0,-239), 53.7 %
16 Lc0 0.25.1 t60-4175 (30x384)     : 3592 515 (+275,=  0,-240), 53.4 %
17 Lc0 0.25.1 t60-4082 (30x384)     : 3589 510 (+270,=  0,-240), 52.9 %
18 Lc0 0.26.0 J90-40 (30x384)       : 3587 511 (+269,=  0,-242), 52.6 %
19 Lc0 0.26.2 PhoenixStein (20x256) : 3585 525 (+275,=  0,-250), 52.4 %
20 Lc0 0.25.1 t40-1541 (20x256)     : 3583 516 (+269,=  0,-247), 52.1 %
21 Allie 0.7 LS 14.3 (20x256)       : 3582 512 (+266,=  0,-246), 52.0 %
22 Lc0 0.25.1 t60-3010 (30x384)     : 3582 514 (+267,=  0,-247), 51.9 %
23 Lc0 0.26.1 64623 (24x320)        : 3576 520 (+266,=  0,-254), 51.2 %
** Stockfish 200418 *************** : 3568 SPCC-Elo *******************
24 Lc0 0.26.1 64208 (24x320)        : 3565 510 (+253,=  0,-257), 49.6 %
25 Allie 0.6 LS 14.3 (20x256)       : 3558 519 (+252,=  0,-267), 48.6 %
26 Lc0 0.25.1 42850 (20x256)        : 3556 522 (+252,=  0,-270), 48.3 %
27 Lc0 0.25.1 63651 (24x320)        : 3554 517 (+248,=  0,-269), 48.0 %
28 Lc0 0.25.1 63851 (24x320)        : 3552 518 (+247,=  0,-271), 47.7 %
28 Lc0 0.25.1 702820 (10x128)       : 3552 518 (+247,=  0,-271), 47.7 %
30 Lc0 0.25.1 t60-3972 (30x384)     : 3550 514 (+244,=  0,-270), 47.5 %
31 Lc0 0.26.0 703810 (10x128)       : 3545 507 (+237,=  0,-270), 46.7 %
32 Lc0 0.26.1 722052 (10x128)       : 3543 512 (+238,=  0,-274), 46.5 %
33 Fat Fritz 1.1 (20x256)           : 3530 523 (+233,=  0,-290), 44.6 %
34 Lc0 0.25.1 63305 (24x320)        : 3530 512 (+228,=  0,-284), 44.5 %
35 Lc0 0.26.1 721051 (10x128)       : 3491 516 (+202,=  0,-314), 39.1 %
36 Lc0 0.25.1 32930 (20x256)        : 3483 515 (+196,=  0,-319), 38.1 %
37 Lc0 0.25.1 714646 (19x256)       : 3479 516 (+194,=  0,-322), 37.6 %
38 Lc0 0.25.1 714435 (19x256)       : 3463 517 (+183,=  0,-334), 35.4 %
39 Lc0 0.26.1 715842 (19x256)       : 3431 528 (+166,=  0,-362), 31.4 %
40 Lc0 0.25.1 11260 (20x256)        : 3408 521 (+149,=  0,-372), 28.6 %

Mention, the number of games is a little bit too high, because the (rare) wins
of Black are doubled in the pgn-file, which is given to ORDO, because of 
Advanced Armageddon Scoring (= a win for Black is 2 points for Black). 
That trick of doubling these games is the only possibility to make
ORDO count a win of Black as 2 points...

Games        : 20630 (finished)

White Wins   : 10517 (51.0 %)
Black Wins   : 10113 (49.0 %)
Draws        : 0 (0.0 %)

Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus Stockfish. Because Lc0 vs. Stockfish is definitly the most interesting head-to-head competition of NN vs. AB-engines. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here

Stockfish (with nnue) vs Lc0 longtime testing

Each testrun 300 games with 150 Noomen lowdraw-openings (selected openings from TCEC superfinals) and 5'+3'' thinking-time (Lc0) / 7.5'+4.5'' (Stockfish). This thinking-time gives a perfect Leela-Ratio of 1.0 on the used PC hardware: i7-8750H 2.6GHz (Hexacore, TurboBoost mode off) Notebook, RTX 2060 GPU. Average game-duration: 20 minutes. Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off), Lc0 (with old 32930 20x256 net) on RTX 2060 mobile: 16000 n/s in starting position.

Hash / NN Cache: 4096 GB Hash for Stockfish / 10000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: 150 Noomen lowdraws openings (J. Noomen selected non-drawish openings out of his TCEC superfinal openings of previous TCEC seasons). Download here

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 5'+3'' and Stockfish 7.5'+4.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 20 minutes.

Download all played games here

Latest update: 2020/09/11 (Stockfish 12 vs Lc0 026.2 J92-130)

Stockfish 12 bmi2 vs Lc0 0.26.2 J92-130:       300 (+ 74,=203,- 23), 58.5 % (+60 Elo)
SF 200823 82215d0fd0df vs Lc0 0.26.1 t60-4619: 300 (+ 85,=199,- 16), 61.5 % (+82 Elo)
SF 200810 112bb1c8cdb5 vs Lc0 0.26.1 LS 15:    300 (+ 78,=196,- 26), 58.7 % (+62 Elo)