Stefan Pohl Computer Chess

private website for chessengine-tests


Lc0 (and other NN-engines) testing

 

Playing conditions:

 

Hardware: i7-8750H (Hexacore) Notebook with RTX 2060 GPU, Windows 10 64bit, 16GB RAM

CPU-Speed: Stockfish with 97% CPU-Speed (to switch off the Intel Turbo Boost): 7.5 MN/s in starting-position, running on 11 threads.

GPU (used by LC Zero): Nvidia RTX 2060 (6GB). LC Zero calculates around 11500 n/s in the starting position (I used the MSI-Afterburner-tool to reduce the speed of the RTX-Card as far as possible) (measured with "go infinte") with Net 32930 (Netsize 20x256), which means a Leela-Ratio (what is Leela Ratio? look here) of 1.3.The Leela-Ratio-value of AlphaZero (used a 20x256 net, too) in the match vs. Stockfish 8 was 1.0 - so 1.3 is a high value, but acceptable.

Hash: 512 MByte for AB-engines and 500.000 size of NNCache for Leela

GUISince 19/09/11: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board), before: LittleBlitzerGUI (draw at 170 moves, resign at -700cp)

TablebasesNone for engines, 5 Syzygy for cutechess-cli

Openings: 250 HERT openings. Download them here

Large Memory Pages: Off

Ponder: Off

Thinking time: 50'' + 500ms (average game-duration: 3 minutes) The thinking-time is not so short, as it seems: Mention, that the AB-engines are running with 5.5 cores, so around 5x more nodes are calculated. Means around 4'+2.5'' thinking-time on singlethread. So, this is still longer, than the 3'+1'' (singlethread), which are used for my Stockfish-testings on the main site!

 

LC Zero Github (information, download Networks and LC Zero Engine): here

Read more about, how LC Zero works, in the LC0-Blog: here

 

Lc0 (or other NN-engine) plays a gauntlet (3000 games) vs. these 6 AB-engines: Stockfish 190622, Houdini 6, Komodo 13.1, Fire 7.1, Ethereal 11.53, Xiphos 0.5.3. All AB-engines running with 11 threads (=5.5 of 6 CPU-cores), 1 thread is for Windows...

 

Last update: 2020/01/05: Lc0 0.23.1 384x30-t40-1573 Net

Next testrun: None. The bullet-NN-testing is discontinued... Check out the excellent new bullet-ratinglist of Andreas Strangmueller instead: here

I decided to discontinue my bullet-NN-testing, because Andreas has a faster hardware and does nearly the same testing, as I did. So, for me, it makes no sense, to continue my testings. Because my hardware is slower (notebooks), it makes much more sense to continue my NN-testings with longer thinking-time. Checkout the new "NN longtime testing"- section of my website...

This section and ratinglist will stay here for 1-2 months and then, I will delete it... So, if you want to download the played games, do it right now!

See the individual statistics of engine-results here

Download all played NN-games here

     Program                         Elo    +    -   Games   Score   Av.Op.  Draws

   1 Lc0 0.23.1 LS 12.2            : 3556   10   10  3000    70.5 %   3395   44.6 %
   2 Lc0 0.23.0 LS 12.1            : 3535   10   10  3000    68.2 %   3395   46.0 %
   3 Lc0 0.22.0 T40B.4-160         : 3533    9    9  3000    67.9 %   3395   47.0 %
   4 Stockfish 190622 bmi2         : 3532    3    3 20000    60.8 %   3450   53.8 %
   5 Lc0 0.21.2 42741              : 3528   10   10  3000    67.5 %   3393   46.7 %
   6 Lc0 0.23.0 58573+             : 3527   10   10  3000    67.2 %   3395   47.3 %
   7 Lc0 0.22.0 J20-460            : 3526    9    9  3000    67.1 %   3395   46.1 %
   8 Lc0 0.22.0 T40B.2-106         : 3525    9    9  3000    67.1 %   3393   45.7 %
   9 Lc0 0.23.1 58613+             : 3524    9    9  3000    66.8 %   3395   45.9 %
  10 Lc0 0.21.3 42850              : 3521    9    9  3000    66.7 %   3393   46.0 %
  11 Lc0 0.21.2 42595              : 3518    9    9  3000    66.3 %   3393   47.0 %
  12 Lc0 0.22.0 49921              : 3516   10   10  3000    65.9 %   3395   47.7 %
  13 Lc0 0.21.2 T40.T8.610         : 3516   10   10  3000    66.1 %   3393   46.0 %
  14 Lc0 0.22.0 LD2+               : 3508    9    9  3000    64.9 %   3395   48.5 %
  15 Lc0 0.22.0 LStein 10.2        : 3496    9    9  3000    63.7 %   3393   46.0 %
  16 Lc0 0.22.0 J13B.2-200         : 3495    9    9  3000    63.3 %   3395   47.7 %
  17 Allie 0.5 LS 11.1             : 3489    9    9  3000    62.5 %   3395   50.3 %
  18 Lc0 0.22.0 LD2                : 3488    9    9  3000    62.6 %   3393   46.9 %
  19 Allie 0.5dev LS 11            : 3486   10   10  3000    62.1 %   3395   52.5 %
  20 Fat Fritz 1.0                 : 3485    9    9  3000    62.0 %   3395   51.3 %
  21 Lc0 0.23.1 384x30-t40-1573    : 3478    9    9  3000    61.2 %   3395   47.7 %
  22 Lc0 0.21.4 32930              : 3467    9    9  3000    60.0 %   3393   50.2 %
  23 Scorpio 3.02 32930            : 3464    9    9  3000    59.3 %   3395   54.4 %
  24 Allie 0.5dev LS 10.2          : 3464    9    9  3000    59.3 %   3395   51.7 %
  25 Lc0 0.22.0 384x30-t40-1207    : 3460    9    9  3000    58.8 %   3395   50.0 %
  26 Houdini 6 pext                : 3450    4    4 21000    49.4 %   3453   54.1 %
  27 Lc0 0.22.0 11260              : 3439    9    9  3000    56.3 %   3393   53.8 %
  28 Lc0 0.22.0 384x30-t40-1097    : 3434    9    9  3000    55.3 %   3395   48.2 %
  29 Komodo 13.1 bmi2              : 3431    4    4 14500    49.1 %   3436   51.7 %
  30 Komodo 13.01 bmi2             : 3422    5    5  9500    47.3 %   3441   51.9 %
  31 Lc0 0.22.0 61211              : 3411    9    9  3000    52.3 %   3395   50.9 %
  32 Lc0 0.22.0 60891              : 3387    9    9  3000    49.0 %   3395   48.3 %
  33 Scorpio 3 NN-Maddex           : 3374    9    9  3000    47.2 %   3395   50.6 %
  34 Fire 7.1 popc                 : 3328    4    4 21000    32.7 %   3459   44.4 %
  35 Xiphos 0.5.3 bmi2             : 3319    4    4 21000    31.6 %   3459   44.7 %
  36 Ethereal 11.53 pext           : 3309    4    4 21000    30.4 %   3459   43.6 %
  37 Lc0 0.22.0 DarkQueen 2.0      : 3182   11   11  3000    23.9 %   3395   33.7 %

 

58613+ settings: CPuct=2.00, FpuValue=0.50, PolicyTemperature=1.50

58573+ settings: CPuct=2.70, FpuValue=0.50, PolicyTemperature=1.53

LD2+ settings: CPuct=2.78, FpuValue=0.43, PolicyTemperature=1.87

Net 42850 was the final Net of the 40xxx learning

Net T40.T8.610 played in TCEC Superfinal Season 15

Net 32930 was the final Net of 30xxx learning

Net 11260 was the final Net of 10xxx learning

 

This rating-list was built out of the gamebase of my Stockfish-testings on the main site and the games, Lc0 plays here in it's testruns. Mention that the conditions of both testings are not exactly the same:

Stockfish-testing: 3'+1'' singlecore, HERT openings

Lc0-testing: 50''+500ms RTX 2060 / Hexacore (means around 4'+2.5'' on singlecore for the AB-engines), HERT Openings.

But mention on the other hand, Lc0 and classical AB-engines cannot be tested with the same conditions, because Lc0 runs on the GPU and works in a completely different way, than AB-engines - we have the Leela-Ratio for comparsion, but even a value of 1.0 does not mean exactly fair or same conditions. So, I believe, it is possible to merge both testings in one rating-list...