Stefan Pohl Computer Chess

private website for chessengine-tests


Lc0 or other GPU-Neural Nets versus KomodoDragon 3 testing

 

Playing conditions:

 

Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM

Cuda version installed: Cuda 11.6

Speed:  Because of my slow hardware and the facts, that (1) the nets of Lc0 getting bigger and bigger and (2) in TCEC the Leela-Ratio is meanwhile around 2.0, I decided not to slow down my RTX 2060 anymore and dont give Lc0 less thinking-time as before (where the Leela-Ratio was 1.0). This leads to a Leela-Ratio of around 2.0, which is an advantage for Lc0 (but my slow hardware is a disadvantage). Additionally I raised the time-increment to 2 seconds (instead of 1 second before), because Lc0 needs more time on a slow hardware and a big net. So the new thinking-time is 2min+2sec.

KomodoDragon plays with 11 Threads (=5.5 cores). Lc0 minibatchsize parameter is set to the best value for each netsize, depending on Lc0's benchmark with backendbench --clippy. For the most (bigger) nets, the best value is 30.

Hash / NN Cache: 2048 GB Hash for KomodoDragom / 1000000 NN-Cachesize for Lc0

GUICutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: UHO_2022_6mvs_+110_+119.pgn. Download my UHO 2022 openings here

Ponder, Large Memory Pages & learning: Off

Thinking time: 2min+2sec. One 600 games-testrun takes around nearly 4 days. 

 

Each Lc0 / Neural Net plays 600 games vs. KomodoDragon 3 with my UHO 2022 openings

 

Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here

 

Latest update: 2022/09/25: Lc0 0.30dev 784968 - new best value. Lc0 binary and net are the same, that were sent to TCEC for playing the premier division in TCEC 23.

Download latest Lc0 dev binaries here

 

Download all played games (games of the old test-setup, too): here

The results of the old test-setup can be seen here

     Program                          Elo    +    -  Games    Score   Av.Op. Draws

   1 Lc0 0.30dev 784968 (20x512)    :   16   18   18   600    52.3%      0   58.3%
   2 Lc0 0.30dev 784010 (20x512)    :   13   20   20   600    51.8%      0   58.3%
   3 Lc0 0.30dev 784765 (20x512)    :    9   18   18   600    51.3%      0   59.7%
   4 KomodoDragon 3 avx2            :    0    5    5  9600    53.6%    -26   57.0%
   5 Lc0 0.29rc1 783467 (20x512)    :   -5   19   19   600    49.3%      0   57.0%
   6 Lc0 0.30dev 803001 (15x512)    :   -7   19   19   600    49.0%      0   58.3%
   7 Lc0 0.29rc1 782922 (20x512)    :   -8   19   19   600    48.8%      0   54.3%
   8 Lc0 0.30dev 803399 (15x512)    :  -13   19   19   600    48.2%      0   60.0%
   9 Lc0 0.30dev 802177 (15x512)    :  -16   20   20   600    47.8%      0   57.8%
  10 Lc0 0.30dev 804287 (15x512)    :  -22   21   21   600    46.9%      0   57.5%
  11 Lc0 0.29rc1 801237 (15x512)    :  -27   18   18   600    46.2%      0   58.3%
  12 Lc0 0.29rc1 611246 (30x384)    :  -29   18   18   600    45.9%      0   57.5%
  13 Lc0 0.29rc1 801663 (15x512)    :  -41   19   19   600    44.2%      0   56.0%
  14 Lc0 0.29rc1 800815 (15x512)    :  -48   19   19   600    43.3%      0   56.2%
  15 Lc0 0.29rc1 791921 (15x192)    :  -64   19   19   600    40.9%      0   52.8%
  16 Lc0 0.29rc1 791810 (15x192)    :  -77   20   20   600    39.2%      0   56.0%
  17 Lc0 0.29rc1 606511 (24x320)    :  -91   20   20   600    37.3%      0   53.3%


Games        : 9600 (finished)

White Wins   : 4064 (42.3 %)
Black Wins   : 67 (0.7 %)
Draws        : 5469 (57.0 %)

 

Below the gamebase recalculated with my Gamepairs Rescorer Batch-Tool. Realizing Vondele's (Stockfish maintainer) idea: "Thinking uniquely in game pairs makes sense with the biased openings used these days. While pentanomial makes sense it is a bit complicated so we could simplify and score game pairs only (not games) as W-L-D (a traditional  score of 2-0, or 1.5-0.5 is just a W)."

   # PLAYER                       :  RATING  ERROR  PLAYED   W    D    L   Score
   1 Lc0 0.30dev 784968 (20x512)  :      29    30     300    82  161   57  54.2%
   2 Lc0 0.30dev 784010 (20x512)  :      26    30     300    80  162   58  53.7%
   3 Lc0 0.30dev 784765 (20x512)  :      20    27     300    76  165   59  52.8%
   4 KomodoDragon 3 avx2          :       0
   5 Lc0 0.29rc1 783467 (20x512)  :     -11    30     300    72  147   81  48.5%
   6 Lc0 0.30dev 803001 (15x512)  :     -15    30     300    59  169   72  47.8%
   7 Lc0 0.29rc1 782922 (20x512)  :     -20    30     300    72  139   89  47.2%
   8 Lc0 0.30dev 803399 (15x512)  :     -26    30     300    55  168   77  46.3%
   9 Lc0 0.30dev 802177 (15x512)  :     -32    30     300    63  147   90  45.5%
  10 Lc0 0.30dev 804287 (15x512)  :     -42    28     300    50  164   86  44.0%
  11 Lc0 0.29rc1 801237 (15x512)  :     -53    28     300    43  169   88  42.5%
  12 Lc0 0.29rc1 611246 (30x384)  :     -55    28     300    43  167   90  42.2%
  13 Lc0 0.29rc1 801663 (15x512)  :     -82    29     300    44  143  113  38.5%
  14 Lc0 0.29rc1 800815 (15x512)  :     -95    28     300    42  137  121  36.8%
  15 Lc0 0.29rc1 791921 (15x192)  :    -133    32     300    25  141  134  31.8%
  16 Lc0 0.29rc1 791810 (15x192)  :    -158    32     300    24  125  151  28.8%
  17 Lc0 0.29rc1 606511 (24x320)  :    -185    32     300    20  115  165  25.8%

You can download my Gamepairs Rescorer Tool right here

 

Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus KomodoDragon. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here