Stefan Pohl Computer Chess

LC0 / Neural Nets versus Stockfish testing

Playing conditions:

Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM

Speed:  Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off): 9000 kn/s, Lc0 (with old 32930 20x256 net): 16000 n/s in starting position. Since Lc0 0.26.3, Lc0 uses Cuda 11.1, which makes it around +37% faster. To keep the Leela-Ratio on 1.0, I slowed down the GPU with the Afterburner-Tool.

Hash / NN Cache: 4096 GB Hash for Stockfish / 5000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: NBSC Advanced Armageddon Noomen 3-moves (250 openings).  Learn more about Advanced Armageddon in the "NBSC Armageddon openings"- section and download the NBSC-Armageddon package right here

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 2'+1'' and Stockfish 3'+1.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 8 minutes, one 500 games-testrun takes around 2.5 days. 

Each Lc0 / Neural Net plays 500 games vs. Stockfish with my new NBSC Advanced Armageddon openings. After the testrun is finished, all games are rescored with my armageddonize_advanced-tool. Means: 

Win for white = 1 point for white
Draw = 1 point for black
Win for black = 2 points for black 

Learn more about my new NBSC Advanced Armageddon openings and the advanced scoring system in the "NBSC Armageddon openings"- section.

Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here

Latest update: 2020/12/05: Lc0 0.26.3 J96-28

(Since Lc0 0.26.3, Lc0 uses Cuda 11.1, which makes it around +37% faster. To keep the Leela-Ratio on 1.0, I slowed down the GPU with the Afterburner-Tool.)

Download all played games (non-armageddonized) here

500 NBSC-Advanced-Armageddon games each testrun (= a win for Black is 2 points for Black and a draw is a 1 point-win for Black). vs. Stockfish 200418 (SPCC-Elo: 3568 (Contempt set to 0) (around +14 Elo stronger than Stockfish 11 (SPCC-Elo: 3554)).

The errorbar of each result is +/- 20 Elo. But mention, that the usage of my NBSC-Armageddon openings spreads the Elo-results around 2.25x wider, than using classical openings for testing(!), so with classical openings, you would need an errorbar of +/- 9 Elo for the same statistical quality of the results (= the rankings of Lc0 nets here). And for an errorbar of +/- 9 elo, you need around 3000 games, not 500, which means 6x more games (and 6x more PC-time)!!

Learn more about that revolution in computerchess in the "NBSC Armageddon openings"- section of my website.

1  Lc0 0.26.3 J96-28 (30x384)       : 3707 517 (+356,=  0,-161), 68.9 %
2  Lc0 0.26.3 J94-80 (30x384)       : 3696 519 (+350,=  0,-169), 67.4 %
3  Lc0 0.26.3 J92-260 (30x384)      : 3689 515 (+343,=  0,-172), 66.6 %
4  Lc0 0.26.3 J92-330 (30x384)      : 3682 519 (+341,=  0,-178), 65.7 %
5  Lc0 0.26.3 65981 (24x320)        : 3668 515 (+329,=  0,-186), 63.9 %
6  Lc0 0.26.3 J92-300 (30x384)      : 3665 513 (+326,=  0,-187), 63.5 %
7  Lc0 0.26.3 66309 (24x320)        : 3662 514 (+324,=  0,-190), 63.0 %
8  Lc0 0.26.2 J92-130 (30x384)      : 3655 521 (+324,=  0,-197), 62.2 %
9  Lc0 0.26.3 65732 (24x320)        : 3648 514 (+315,=  0,-199), 61.3 %
10 Lc0 0.26.3 65536 (24x320)        : 3648 514 (+315,=  0,-199), 61.3 %
11 Lc0 0.24.1 LS 14.3 (20x256)      : 3644 513 (+311,=  0,-202), 60.6 %
12 Lc0 0.25.1 LS 15 (20x256)        : 3643 512 (+310,=  0,-202), 60.5 %
13 Lc0 0.26.3 65411 (24x320)        : 3641 519 (+313,=  0,-206), 60.3 %
14 Lc0 0.26.2 J92-160 (30x384)      : 3635 511 (+304,=  0,-207), 59.5 %
15 Lc0 0.26.2 T60B.7-105 (24x320)   : 3634 519 (+308,=  0,-211), 59.3 %
16 Lc0 0.24.1 LS 14.2 (20x256)      : 3633 520 (+308,=  0,-212), 59.2 %
17 Lc0 0.25.1 LS 15 Kayra4          : 3624 513 (+297,=  0,-216), 57.9 %
18 Lc0 0.26.1 t60-4619 (30x384)     : 3622 522 (+301,=  0,-221), 57.7 %
19 Lc0 0.26.2 J92-205 (30x384)      : 3618 511 (+292,=  0,-219), 57.1 %
20 Lc0 0.25.1 3972_20k_tcec (30x384): 3617 514 (+293,=  0,-221), 57.0 %
21 Lc0 0.26.2 65100 (24x320)        : 3616 512 (+291,=  0,-221), 56.8 %
22 Lc0 0.26.1 J92-100 (30x384)      : 3609 510 (+285,=  0,-225), 55.9 %
23 Lc0 0.26.1 t60-4585 (30x384)     : 3605 514 (+284,=  0,-230), 55.3 %
24 Lc0 0.25.1 sv-1810 (20x256)      : 3599 514 (+280,=  0,-234), 54.5 %
25 Lc0 0.25.1 t60-4175_mlh (30x384) : 3594 516 (+277,=  0,-239), 53.7 %
26 Lc0 0.25.1 t60-4175 (30x384)     : 3592 515 (+275,=  0,-240), 53.4 %
27 Lc0 0.25.1 t60-4082 (30x384)     : 3589 510 (+270,=  0,-240), 52.9 %
28 Lc0 0.26.0 J90-40 (30x384)       : 3587 511 (+269,=  0,-242), 52.6 %
29 Lc0 0.26.2 PhoenixStein (20x256) : 3585 525 (+275,=  0,-250), 52.4 %
30 Lc0 0.25.1 t40-1541 (20x256)     : 3583 516 (+269,=  0,-247), 52.1 %
31 Allie 0.7 LS 14.3 (20x256)       : 3582 512 (+266,=  0,-246), 52.0 %
32 Lc0 0.25.1 t60-3010 (30x384)     : 3582 514 (+267,=  0,-247), 51.9 %
33 Lc0 0.26.1 64623 (24x320)        : 3576 520 (+266,=  0,-254), 51.2 %
** Stockfish 200418 *************** : 3568 SPCC-Elo *******************
34 Lc0 0.26.1 64208 (24x320)        : 3565 510 (+253,=  0,-257), 49.6 %
35 Allie 0.6 LS 14.3 (20x256)       : 3558 519 (+252,=  0,-267), 48.6 %
36 Lc0 0.25.1 42850 (20x256)        : 3556 522 (+252,=  0,-270), 48.3 %
37 Lc0 0.25.1 63651 (24x320)        : 3554 517 (+248,=  0,-269), 48.0 %
38 Lc0 0.25.1 63851 (24x320)        : 3552 518 (+247,=  0,-271), 47.7 %
38 Lc0 0.25.1 702820 (10x128)       : 3552 518 (+247,=  0,-271), 47.7 %
40 Lc0 0.25.1 t60-3972 (30x384)     : 3550 514 (+244,=  0,-270), 47.5 %
41 Lc0 0.26.2 722641 (10x128)       : 3546 518 (+243,=  0,-275), 46.9 %
42 Lc0 0.26.0 703810 (10x128)       : 3545 507 (+237,=  0,-270), 46.7 %
43 Lc0 0.26.1 722052 (10x128)       : 3543 512 (+238,=  0,-274), 46.5 %
44 Lc0 0.26.3 730372 (14x128)       : 3536 510 (+232,=  0,-278), 45.5 %
45 Fat Fritz 1.1 (20x256)           : 3530 523 (+233,=  0,-290), 44.6 %
46 Lc0 0.25.1 63305 (24x320)        : 3530 512 (+228,=  0,-284), 44.5 %
47 Lc0 0.26.3 730262 (14x128)       : 3528 515 (+228,=  0,-287), 44.3 %
48 Lc0 0.26.3 730517 (14x128)       : 3525 520 (+228,=  0,-292), 43.8 %
49 Lc0 0.26.1 721051 (10x128)       : 3491 516 (+202,=  0,-314), 39.1 %
50 Lc0 0.25.1 32930 (20x256)        : 3483 515 (+196,=  0,-319), 38.1 %
51 Lc0 0.25.1 714646 (19x256)       : 3479 516 (+194,=  0,-322), 37.6 %
52 Lc0 0.26.3 730164 (14x128)       : 3475 513 (+190,=  0,-323), 37.0 %
53 Lc0 0.25.1 714435 (19x256)       : 3463 517 (+183,=  0,-334), 35.4 %
54 Lc0 0.26.1 715842 (19x256)       : 3431 528 (+166,=  0,-362), 31.4 %
55 Lc0 0.25.1 11260 (20x256)        : 3408 521 (+149,=  0,-372), 28.6 %
56 Lc0 0.26.3 Bad Gyal 9XL (20x128) : 3367 528 (+127,=  0,-401), 24.1 %

Mention, the number of games is a little bit too high, because the (rare) wins
of Black are doubled in the pgn-file, which is given to ORDO, because of 
Advanced Armageddon Scoring (= a win for Black is 2 points for Black). 
That trick of doubling these games is the only possibility to make
ORDO count a win of Black as 2 points...

Games        : 28893 (finished)

White Wins   : 14832 (51.3 %)
Black Wins   : 14061 (48.7 %)
Draws        : 0 (0.0 %)

Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus Stockfish. Because Lc0 vs. Stockfish is definitly the most interesting head-to-head competition of NN vs. AB-engines. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here

Stockfish (with nnue) vs Lc0 longtime testing

Each testrun 300 games with 150 Noomen lowdraw-openings (selected openings from TCEC superfinals) and 5'+3'' thinking-time (Lc0) / 7.5'+4.5'' (Stockfish). This thinking-time gives a perfect Leela-Ratio of 1.0 on the used PC hardware: i7-8750H 2.6GHz (Hexacore, TurboBoost mode off) Notebook, RTX 2060 GPU. Average game-duration: 20 minutes. Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off), Lc0 (with old 32930 20x256 net) on RTX 2060 mobile: 16000 n/s in starting position.

Hash / NN Cache: 4096 GB Hash for Stockfish / 10000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: 150 Noomen lowdraws openings (J. Noomen selected non-drawish openings out of his TCEC superfinal openings of previous TCEC seasons). Download here

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 5'+3'' and Stockfish 7.5'+4.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 20 minutes.

Download all played games here

Latest update: 2020/11/21 (Huge (450 games, 8 days) RoundRobin tournament of the 3 top-engines, using my new Unbalanced Human Openings (UHO 1.0). 

See some short and spectacular wins of this match directly here on the website in the "View SF vs Lc0 games"- section!

******************************************************************
***** 3 Top Engines RoundRobin tournament, UHO openings **********
******************************************************************
1 Stockfish 201022 bmi2       : 3656 300 (+111,=149,- 40), 61.8 %

KomodoDragon 1.0 x64          : 150 (+ 58,= 73,- 19), 63.0 %
Lc0 0.26.3 J92-260 (30x384)   : 150 (+ 53,= 76,- 21), 60.7 %

2 Lc0 0.26.3 J92-260 (30x384) : 3573 300 (+ 55,=156,- 89), 44.3 %

Stockfish 201022 bmi2         : 150 (+ 21,= 76,- 53), 39.3 %
KomodoDragon 1.0 x64          : 150 (+ 34,= 80,- 36), 49.3 %

3 KomodoDragon 1.0 x64        : 3571 300 (+ 55,=153,- 92), 43.8 %

Stockfish 201022 bmi2         : 150 (+ 19,= 73,- 58), 37.0 %
Lc0 0.26.3 J92-260 (30x384)   : 150 (+ 36,= 80,- 34), 50.7 %
******************************************************************
Stockfish 201022 bmi2 vs Lc0 0.26.3 J92-260   : 300 (+ 75,=207,- 18), 59.5 % (+67 Elo)
Stockfish 200928 bmi2 vs Lc0 0.26.3rc2 J92-190: 300 (+ 68,=215,- 17), 58.5 % (+60 Elo)
Stockfish 12 bmi2 vs Lc0 0.26.2 J92-130:        300 (+ 74,=203,- 23), 58.5 % (+60 Elo)
SF 200823 82215d0fd0df vs Lc0 0.26.1 t60-4619:  300 (+ 85,=199,- 16), 61.5 % (+82 Elo)
SF 200810 112bb1c8cdb5 vs Lc0 0.26.1 LS 15:     300 (+ 78,=196,- 26), 58.7 % (+62 Elo)