Stefan Pohl Computer Chess

Lc0 or other GPU-Neural Nets versus Stockfish testing

Playing conditions:

Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM

Cuda version installed: Cuda 11.4

Speed:  Stockfish 8 (running on 11 hyperthreading-threads, Intel Turbo-Mode off): 9000 kn/s, Lc0 (with old 32930 20x256 net): 16000 n/s in starting position. Since Lc0 0.26.3, Lc0 uses Cuda 11.1, which makes it around +37% faster. To keep the Leela-Ratio on 1.0, I slowed down the GPU with the Afterburner-Tool. In Lc0 MiniBatchSize is set to 80 (which is the fastest configuration, I could find)

Hash / NN Cache: 4096 GB Hash for Stockfish / 5000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: Unbalanced Human Openings V3 6moves with eval [+0.90;+0.99]. Learn more in the "Anti Draw Openings"- section.

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 2'+1'' and Stockfish 3'+1.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 8 minutes, one 750 games-testrun takes around 4 days. 

Each Lc0 / Neural Net plays 750 games vs. Stockfish 13 with my new Unbalanced Human Openings V3

Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here

Latest update: 2021/10/18: Lc0 0.28.0 610226 Next testrun: Lc0 0.28.0 754042 (final net of T75-run)

Download all played games (games of the old test-setup, too): here

The results of the old test-setup can be seen here

     Program                            Elo    +    -   Games   Score   Av.Op.  Draws

   1 Stockfish 13 bmi2                : 3723    3    3 24750    65.0 %   3612   51.7 %
   2 Lc0 0.28.0 610062 (30x384)       : 3667   18   18   750    42.1 %   3723   55.9 %
   3 Lc0 0.27.0 69722 (30x384)        : 3666   18   18   750    41.9 %   3723   55.3 %
   4 Lc0 0.28.0 610226 (30x384)       : 3663   17   17   750    41.5 %   3723   57.5 %
   5 Lc0 0.27.0 69722vf20 (30x384)    : 3662   18   18   750    41.4 %   3723   57.2 %
   6 Lc0 0.28.0 610176 (30x384)       : 3662   18   18   750    41.3 %   3723   55.5 %
   7 Lc0 0.27.0 69626 (30x384)        : 3661   19   19   750    41.3 %   3723   54.5 %
   8 Lc0 0.27.0 68695 (30x384)        : 3661   18   18   750    41.2 %   3723   56.8 %
   9 Lc0 0.28.0 609973 (30x384)       : 3657   18   18   750    40.7 %   3723   56.3 %
  10 Lc0 0.28.0 610035 (30x384)       : 3654   18   18   750    40.3 %   3723   52.3 %
  11 Lc0 0.27.0 69146 (30x384)        : 3648   18   18   750    39.4 %   3723   54.0 %
  12 Lc0 0.27.0 69363 (30x384)        : 3648   18   18   750    39.4 %   3723   52.9 %
  13 Ceres 0.93 69722                 : 3646   18   18   750    39.2 %   3723   53.1 %
  14 Ceres 0.91b 69722                : 3646   18   18   750    39.1 %   3723   52.4 %
  15 Lc0 0.27.0 J94-100 (30x384)      : 3646   18   18   750    39.1 %   3723   51.3 %
  16 Ceres 0.90rc1 69626              : 3645   18   18   750    39.0 %   3723   52.9 %
  17 Lc0 0.28.0 752390 (15x192)       : 3616   19   19   750    35.1 %   3723   50.8 %
  18 Lc0 0.28.0 753690 (15x192)       : 3615   18   18   750    35.1 %   3723   54.7 %
  19 Lc0 0.28.0 752771 (15x192)       : 3612   19   19   750    34.7 %   3723   52.4 %
  20 Lc0 0.28.0 753444 (15x192)       : 3611   19   19   750    34.6 %   3723   52.9 %
  21 Lc0 0.27.0 751735 (15x192)       : 3601   19   19   750    33.3 %   3723   52.7 %
  22 Lc0 0.27.0 752115 (15x192)       : 3600   18   18   750    33.2 %   3723   53.3 %
  23 Lc0 0.27.0 LS 15 (20x256)        : 3597   19   19   750    32.8 %   3723   50.4 %
  24 Lc0 0.27.0 751666 (15x192)       : 3597   18   18   750    32.8 %   3723   49.3 %
  25 Lc0 0.27.0 751605 (15x192)       : 3595   18   18   750    32.5 %   3723   51.3 %
  26 Lc0 0.27.0 751236 (15x192)       : 3575   19   19   750    30.1 %   3723   49.2 %
  27 Lc0 0.27.0 J104.1-30 (10x128)    : 3565   20   20   750    28.9 %   3723   49.2 %
  28 Lc0 0.27.0 744548 (10x128)       : 3564   19   19   750    28.8 %   3723   48.5 %
  29 Lc0 0.27.0 744706 (10x128)       : 3563   19   19   750    28.7 %   3723   49.6 %
  30 Lc0 0.27.0 743947 (10x128)       : 3558   19   19   750    28.1 %   3723   48.4 %
  31 Lc0 0.27.0 744204 (10x128)       : 3555   20   20   750    27.7 %   3723   46.9 %
  32 Lc0 0.27.0 743665 (10x128)       : 3552   19   19   750    27.3 %   3723   48.0 %
  33 Lc0 0.27.0 703810 (10x128)       : 3546   20   20   750    26.7 %   3723   46.0 %
  34 ClassicAra 0.9.5                 : 3441   24   24   750    16.7 %   3723   33.3 %

Games        : 24750 (finished)

White Wins   : 11302 (45.7 %)
Black Wins   : 661 (2.7 %)
Draws        : 12787 (51.7 %)

Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus Stockfish. Because Lc0 vs. Stockfish is definitly the most interesting head-to-head competition of NN vs. AB-engines. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here

Stockfish vs Lc0 longtime testing ("SuFi for the poor")

Each testrun 300 games with 150 Noomen lowdraw-openings (selected openings from TCEC superfinals) and 5'+3'' thinking-time (Lc0) / 7.5'+4.5'' (Stockfish). This thinking-time gives a perfect Leela-Ratio of 1.0 on the used PC hardware: i7-8750H 2.6GHz (Hexacore, TurboBoost mode off) Notebook, RTX 2060 GPU. Average game-duration: 20 minutes. Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off) 9000 kn/s, Lc0 (with old 32930 20x256 net) on RTX 2060 mobile: 16000 n/s in starting position.

Hash / NN Cache: 4096 GB Hash for Stockfish / 10000000 NN-Cachesize for Lc0

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: Noomen lowdraws openings (J. Noomen selected non-drawish openings out of his TCEC Superfinal openings of previous TCEC seasons). Download here

Ponder, Large Memory Pages & learning: Off

Thinking time: Lc0 5'+3'' and Stockfish 7.5'+4.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 20 minutes.

Download all played games here

Latest update: 2021/10/02: Stockfish 210915 vs KomodoDragon 2.5 (A disappointing result: A very clear loss of KomodoDragon 2.5, more clear, than I expected.)

See some short and spectacular wins of this match directly here on the website in the "View SF vs Lc0 games"- section!

Stockfish 210915 bmi2 vs KomodoDragon 2.5 x64 : 300 (+ 90,=192,- 18), 62.0 % (+85 Elo)
Stockfish 210915 bmi2 vs Lc0 0.28.0 610062    : 300 (+ 94,=193,- 13), 63.5 % (+96 Elo)
Stockfish 210713 bmi2 vs Lc0 0.28.0-rc1 69626 : 300 (+ 76,=204,- 20), 59.3 % (+66 Elo)
Stockfish 210619 bmi2 vs Lc0 0.28.0-rc1 69146 : 300 (+ 66,=224,- 10), 59.3 % (+66 Elo)
Stockfish 210415 bmi2 vs Lc0 0.27.0 68695     : 300 (+ 62,=218,- 20), 57.0 % (+49 Elo)
Stockfish 210406 bmi2 vs Lc0 0.27.0 68002     : 300 (+ 60,=219,- 21), 56.5 % (+45 Elo)
Stockfish 210226 bmi2 vs Lc0 0.27.0 67741     : 300 (+ 75,=205,- 20), 59.2 % (+65 Elo)
Stockfish 201225 bmi2 vs Lc0 0.26.3 66680     : 300 (+ 60,=223,- 17), 57.2 % (+50 Elo)
Stockfish 201022 bmi2 vs Lc0 0.26.3 J92-260   : 300 (+ 75,=207,- 18), 59.5 % (+67 Elo)
Stockfish 200928 bmi2 vs Lc0 0.26.3rc2 J92-190: 300 (+ 68,=215,- 17), 58.5 % (+60 Elo)
Stockfish 12 bmi2 vs Lc0 0.26.2 J92-130       : 300 (+ 74,=203,- 23), 58.5 % (+60 Elo)
SF 200823 82215d0fd0df vs Lc0 0.26.1 t60-4619 : 300 (+ 85,=199,- 16), 61.5 % (+82 Elo)
SF 200810 112bb1c8cdb5 vs Lc0 0.26.1 LS 15    : 300 (+ 78,=196,- 26), 58.7 % (+62 Elo)