Stefan Pohl Computer Chess

Lc0 or other GPU-Neural Nets versus Stockfish testing

Playing conditions:

Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM

Cuda version installed: Cuda 11.5

Speed:  Stockfish 8 (running on 11 hyperthreading-threads, Intel Turbo-Mode off): 9000 kn/s, Lc0 (with old 32930 20x256 net): 16000 n/s in starting position. Since Lc0 0.26.3, Lc0 uses Cuda 11.1, which makes it around +37% faster. To keep the Leela-Ratio on 1.0, I slowed down the GPU with the Afterburner-Tool. In Lc0 MiniBatchSize is set to 80 (which is the fastest configuration, I could find)

Hash / NN Cache: 4096 GB Hash for Stockfish / 5000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: Unbalanced Human Openings V3 6moves with eval [+0.90;+0.99]. Learn more in the "Anti Draw Openings"- section.

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 2'+1'' and Stockfish 3'+1.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 8 minutes, one 750 games-testrun takes around 4 days. 

Each Lc0 / Neural Net plays 750 games vs. Stockfish 13 with my new Unbalanced Human Openings V3

Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here

Latest update: 2022/01/15: Lc0 0.28.2 T75_2400k (20x256)

Download all played games (games of the old test-setup, too): here

The results of the old test-setup can be seen here

     Program                            Elo    +    -   Games   Score   Av.Op.  Draws

   1 Stockfish 13 bmi2                : 3723    3    3 25500    62.9 %   3628   53.0 %
   2 Lc0 0.28.0 610062 (30x384)       : 3667   17   17   750    42.1 %   3723   55.9 %
   3 Lc0 0.27.0 69722 (30x384)        : 3666   18   18   750    41.9 %   3723   55.3 %
   4 Lc0 0.28.0 610226 (30x384)       : 3663   17   17   750    41.5 %   3723   57.5 %
   5 Lc0 0.27.0 69722vf20 (30x384)    : 3662   18   18   750    41.4 %   3723   57.2 %
   6 Lc0 0.28.0 610176 (30x384)       : 3662   18   18   750    41.3 %   3723   55.5 %
   7 Lc0 0.27.0 69626 (30x384)        : 3661   18   18   750    41.3 %   3723   54.5 %
   8 Lc0 0.27.0 68695 (30x384)        : 3661   18   18   750    41.2 %   3723   56.8 %
   9 Lc0 0.28.2 610889 (30x384)       : 3660   18   18   750    41.1 %   3723   56.7 %
  10 Lc0 0.28.0 609973 (30x384)       : 3657   18   18   750    40.7 %   3723   56.3 %
  11 Lc0 0.28.0 610594 (30x384)       : 3656   18   18   750    40.6 %   3723   57.2 %
  12 Lc0 0.28.2 611216 (30x384)       : 3656   18   18   750    40.5 %   3723   55.2 %
  13 Lc0 0.28.0 610428 (30x384)       : 3656   17   17   750    40.5 %   3723   56.8 %
  14 Lc0 0.28.0 610035 (30x384)       : 3654   18   18   750    40.3 %   3723   52.3 %
  15 Lc0 0.28.0 mg1670k (40x512)      : 3651   19   19   750    39.9 %   3723   54.4 %
  16 Lc0 0.28.0 mg2000k (40x512)      : 3649   19   19   750    39.5 %   3723   54.3 %
  17 Lc0 0.27.0 69146 (30x384)        : 3648   18   18   750    39.4 %   3723   54.0 %
  18 Lc0 0.27.0 69363 (30x384)        : 3648   18   18   750    39.4 %   3723   52.9 %
  19 Ceres 0.94 610062                : 3647   18   18   750    39.3 %   3723   54.8 %
  20 Ceres 0.93 69722                 : 3646   17   17   750    39.2 %   3723   53.1 %
  21 Lc0 0.27.0 J94-100 (30x384)      : 3646   18   18   750    39.1 %   3723   51.3 %
  22 Ceres 0.91b 69722                : 3646   18   18   750    39.1 %   3723   52.4 %
  23 Ceres 0.90rc1 69626              : 3645   18   18   750    39.0 %   3723   52.9 %
  24 Lc0 0.28.0 780023 (40x512)       : 3644   18   18   750    38.9 %   3723   55.6 %
  25 Lc0 0.28.0 754042 (15x192)       : 3622   18   18   750    36.0 %   3723   54.1 %
  26 Lc0 0.28.0 761187 (15x192)       : 3616   18   18   750    35.1 %   3723   54.0 %
  27 Lc0 0.28.2 771567 (15x192)       : 3605   18   18   750    33.7 %   3723   49.6 %
  28 Lc0 0.28.2 T75_2400k (20x256)    : 3602   19   19   750    33.4 %   3723   51.9 %
  29 Lc0 0.27.0 LS 15 (20x256)        : 3597   19   19   750    32.8 %   3723   50.4 %
  30 Lc0 0.28.2 771126 (15x192)       : 3593   18   18   750    32.2 %   3723   52.7 %
  31 Lc0 0.28.0 770578 (15x192)       : 3567   20   20   750    29.1 %   3723   49.7 %
  32 Lc0 0.27.0 J104.1-30 (10x128)    : 3565   19   19   750    28.9 %   3723   49.2 %
  33 Lc0 0.27.0 744548 (10x128)       : 3564   19   19   750    28.8 %   3723   48.5 %
  34 Lc0 0.27.0 703810 (10x128)       : 3546   20   20   750    26.7 %   3723   46.0 %
  35 ClassicAra 0.9.5                 : 3441   23   23   750    16.7 %   3723   33.3 %

Games        : 25500 (finished)

White Wins   : 11462 (44.9 %)
Black Wins   : 521 (2.0 %)
Draws        : 13517 (53.0 %)

Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus Stockfish. Because Lc0 vs. Stockfish is definitly the most interesting head-to-head competition of NN vs. AB-engines. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here

Stockfish vs Lc0 longtime testing ("SuFi for the poor")

Each testrun 300 games with 150 Noomen lowdraw-openings (selected openings from TCEC superfinals) and 5'+3'' thinking-time (Lc0) / 7.5'+4.5'' (Stockfish). This thinking-time gives a perfect Leela-Ratio of 1.0 on the used PC hardware: i7-8750H 2.6GHz (Hexacore, TurboBoost mode off) Notebook, RTX 2060 GPU. Average game-duration: 20 minutes. Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off) 9000 kn/s, Lc0 (with old 32930 20x256 net) on RTX 2060 mobile: 16000 n/s in starting position.

Hash / NN Cache: 4096 GB Hash for Stockfish / 10000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: Noomen lowdraws openings (J. Noomen selected non-drawish openings out of his TCEC Superfinal openings of previous TCEC seasons). Download here

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 5'+3'' and Stockfish 7.5'+4.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 20 minutes.

Download all played games here

Latest update: 2021/10/28: KomodoDragon 2.5 vs Lc0 0.28.0 610062 (Another disappointing result of KomodoDragon 2.5: -89 Elo weaker Elo-performance vs. Lc0 0.28.0 610062, than Stockfish 210915 (+7 Elo to +96 Elo)) - but the results fits very well to the weak result of KomodoDragon 2.5 vs. Stockfish 210915 itself (-85 Elo) (only 4 Elo performance difference, not bad for 300 games, only)...

See some short and spectacular wins of this match directly here on the website in the "View SF vs Lc0 games"- section!

KomodoDragon 2.5 x64 vs Lc0 0.28.0 610062     : 300 (+ 61,=184,- 55), 51.0 % ( +7 Elo)
Stockfish 210915 bmi2 vs KomodoDragon 2.5 x64 : 300 (+ 90,=192,- 18), 62.0 % (+85 Elo)
Stockfish 210915 bmi2 vs Lc0 0.28.0 610062    : 300 (+ 94,=193,- 13), 63.5 % (+96 Elo)
Stockfish 210713 bmi2 vs Lc0 0.28.0-rc1 69626 : 300 (+ 76,=204,- 20), 59.3 % (+66 Elo)
Stockfish 210619 bmi2 vs Lc0 0.28.0-rc1 69146 : 300 (+ 66,=224,- 10), 59.3 % (+66 Elo)
Stockfish 210415 bmi2 vs Lc0 0.27.0 68695     : 300 (+ 62,=218,- 20), 57.0 % (+49 Elo)
Stockfish 210406 bmi2 vs Lc0 0.27.0 68002     : 300 (+ 60,=219,- 21), 56.5 % (+45 Elo)
Stockfish 210226 bmi2 vs Lc0 0.27.0 67741     : 300 (+ 75,=205,- 20), 59.2 % (+65 Elo)
Stockfish 201225 bmi2 vs Lc0 0.26.3 66680     : 300 (+ 60,=223,- 17), 57.2 % (+50 Elo)
Stockfish 201022 bmi2 vs Lc0 0.26.3 J92-260   : 300 (+ 75,=207,- 18), 59.5 % (+67 Elo)
Stockfish 200928 bmi2 vs Lc0 0.26.3rc2 J92-190: 300 (+ 68,=215,- 17), 58.5 % (+60 Elo)
Stockfish 12 bmi2 vs Lc0 0.26.2 J92-130       : 300 (+ 74,=203,- 23), 58.5 % (+60 Elo)
SF 200823 82215d0fd0df vs Lc0 0.26.1 t60-4619 : 300 (+ 85,=199,- 16), 61.5 % (+82 Elo)
SF 200810 112bb1c8cdb5 vs Lc0 0.26.1 LS 15    : 300 (+ 78,=196,- 26), 58.7 % (+62 Elo)