Stefan Pohl Computer Chess

private website for chessengine-tests


Lc0 or other GPU-Neural Nets versus KomodoDragon 3 testing

 

Playing conditions:

 

Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM

Cuda version installed: Cuda 11.7

Speed:  Because of my slow hardware and the facts, that (1) the nets of Lc0 getting bigger and bigger and (2) in TCEC the Leela-Ratio is meanwhile around 2.0, I decided not to slow down my RTX 2060 anymore and dont give Lc0 less thinking-time as before (where the Leela-Ratio was 1.0). This leads to a Leela-Ratio of around 2.0, which is an advantage for Lc0 (but my slow hardware is a disadvantage). Additionally I raised the time-increment to 2 seconds (instead of 1 second before), because Lc0 needs more time on a slow hardware and a big net. So the new thinking-time is 2min+2sec.

KomodoDragon plays with 11 Threads (=5.5 cores). Lc0 minibatchsize parameter is set to the best value for each netsize, depending on Lc0's benchmark with backendbench --clippy. For the most (bigger) nets, the best value is 30.

Hash / NN Cache: 2048 GB Hash for KomodoDragom / 1000000 NN-Cachesize for Lc0

GUICutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: UHO_2022_6mvs_+110_+119.pgn. Download my UHO 2022 openings here

Ponder, Large Memory Pages & learning: Off

Thinking time: 2min+2sec. One 600 games-testrun takes around nearly 4 days. 

 

Each Lc0 / Neural Net plays 600 games vs. KomodoDragon 3 with my UHO 2022 openings

 

Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here

 

Latest update: 2022/11/19: Lc0 0.30dev 806488 - clear regression (perhaps the new binary is weaker?)

Download latest Lc0 dev binaries here

 

Download all played games (games of the old test-setup, too): here

The results of the old test-setup can be seen here

     Program                          Elo    +    -  Games    Score   Av.Op. Draws

   1 Lc0 0.30dev 784968 (20x512)    :   16   19   19   600    52.3%      0   58.3%
   2 Lc0 0.30dev 806012 (15x512)    :   16   18   18   600    52.3%      0   63.2%
   3 Lc0 0.30dev 784010 (20x512)    :   13   19   19   600    51.8%      0   58.3%
   4 Lc0 0.30dev 805123 (15x512)    :    9   20   20   600    51.3%      0   61.0%
   5 Lc0 0.30dev 784765 (20x512)    :    9   18   18   600    51.3%      0   59.7%
   6 KomodoDragon 3 avx2            :    0    4    4 11400    52.9%    -21   57.6%
   7 Lc0 0.29rc1 783467 (20x512)    :   -5   18   18   600    49.3%      0   57.0%
   8 Lc0 0.30dev 803001 (15x512)    :   -7   20   20   600    49.0%      0   58.3%
   9 Lc0 0.29rc1 782922 (20x512)    :   -8   19   19   600    48.8%      0   54.3%
  10 Lc0 0.30dev 806488 (15x512)    :  -11   20   20   600    48.4%      0   58.2%
  11 Lc0 0.30dev 803399 (15x512)    :  -13   19   19   600    48.2%      0   60.0%
  12 Lc0 0.30dev 802177 (15x512)    :  -16   19   19   600    47.8%      0   57.8%
  13 Lc0 0.30dev 804287 (15x512)    :  -22   20   20   600    46.9%      0   57.5%
  14 Lc0 0.29rc1 801237 (15x512)    :  -27   19   19   600    46.2%      0   58.3%
  15 Lc0 0.29rc1 611246 (30x384)    :  -29   19   19   600    45.9%      0   57.5%
  16 Lc0 0.29rc1 801663 (15x512)    :  -41   21   21   600    44.2%      0   56.0%
  17 Lc0 0.29rc1 800815 (15x512)    :  -48   20   20   600    43.3%      0   56.2%
  18 Lc0 0.29rc1 791921 (15x192)    :  -64   19   19   600    40.9%      0   52.8%
  19 Lc0 0.29rc1 791810 (15x192)    :  -77   18   18   600    39.2%      0   56.0%
  20 Lc0 0.29rc1 606511 (24x320)    :  -91   20   20   600    37.3%      0   53.3%


Games        : 11400 (finished)

White Wins   : 4764 (41.8 %)
Black Wins   : 73 (0.6 %)
Draws        : 6563 (57.6 %)

 

Below the gamebase recalculated with my Gamepairs Rescorer Batch-Tool. Realizing Vondele's (Stockfish maintainer) idea: "Thinking uniquely in game pairs makes sense with the biased openings used these days. While pentanomial makes sense it is a bit complicated so we could simplify and score game pairs only (not games) as W-L-D (a traditional  score of 2-0, or 1.5-0.5 is just a W)."

   # PLAYER                       :  RATING  ERROR  PLAYED   W    D    L   Score
   1 Lc0 0.30dev 806012 (15x512)  :      32    30     300    76  175   49  54.5%
   2 Lc0 0.30dev 784968 (20x512)  :      29    30     300    82  161   57  54.2%
   3 Lc0 0.30dev 784010 (20x512)  :      26    30     300    80  162   58  53.7%
   4 Lc0 0.30dev 784765 (20x512)  :      20    27     300    76  165   59  52.8%
   5 Lc0 0.30dev 805123 (15x512)  :      19    30     300    74  168   58  52.7%
   6 KomodoDragon 3 avx2          :       0
   7 Lc0 0.29rc1 783467 (20x512)  :     -11    30     300    72  147   81  48.5%
   8 Lc0 0.30dev 803001 (15x512)  :     -15    30     300    59  169   72  47.8%
   9 Lc0 0.29rc1 782922 (20x512)  :     -20    30     300    72  139   89  47.2%
  10 Lc0 0.30dev 806488 (15x512)  :     -22    30     300    61  159   80  46.8%
  11 Lc0 0.30dev 803399 (15x512)  :     -26    30     300    55  168   77  46.3%
  12 Lc0 0.30dev 802177 (15x512)  :     -32    30     300    63  147   90  45.5%
  13 Lc0 0.30dev 804287 (15x512)  :     -42    28     300    50  164   86  44.0%
  14 Lc0 0.29rc1 801237 (15x512)  :     -53    28     300    43  169   88  42.5%
  15 Lc0 0.29rc1 611246 (30x384)  :     -55    28     300    43  167   90  42.2%
  16 Lc0 0.29rc1 801663 (15x512)  :     -82    29     300    44  143  113  38.5%
  17 Lc0 0.29rc1 800815 (15x512)  :     -95    28     300    42  137  121  36.8%
  18 Lc0 0.29rc1 791921 (15x192)  :    -133    32     300    25  141  134  31.8%
  19 Lc0 0.29rc1 791810 (15x192)  :    -158    32     300    24  125  151  28.8%
  20 Lc0 0.29rc1 606511 (24x320)  :    -185    32     300    20  115  165  25.8%

You can download my Gamepairs Rescorer Tool right here

 

Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus KomodoDragon. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here