Stefan Pohl Computer Chess

Stockfish Regression Testing

Latest testrun: Stockfish 220108
Reference point (opponent) is the latest official SF-release (Stockfish 14.1 right now).
Each SF-dev version plays 20000 games versus this engine

Hardware: AMD Ryzen 3900 12-core (24 threads) notebook with 32GB RAM. 20 games are played simultaneously

Speed: Singlethread, TurboBoost-mode switched off, chess starting position: Stockfish 11: 1.3 mn/s, Komodo 14: 1.1 mn/s

Hash: 128MB per engine

GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)

Tablebases: None for engines, 5 Syzygy for cutechess-cli

Openings: Because meanwhile the high draw-rates made it impossible to measure Elo-progress in regression-tests, here my UHO_V3_8mvs_big_+090_+119 openings are used (part of my Anti Draw openings download). These openings have the similar eval-interval of KomodoDragon 1.0 [+0.90;+1.19] than the UHO_XXL openings, which are used in the Stockfish framework for development. But UHO V3 openings were evaluated with around 50x (!!!) more nodes per endposition, than the UHO_XXL openings, because the UHO V3 files are much smaller.

Ponder, Large Memory Pages & learning: Off

Thinking time: 30 sec + 300 ms per game/engine (average game-duration: around  1.5 minutes). One 20000 games-testrun takes about 30 hours.The version-numbers of the Stockfish engines are the date of the latest patch, which was included in the Stockfish sourcecode, not the release-date of the engine-file, written backwards (year,month,day) (example: 200807 = August, 7, 2020). The used SF compile is the AVX2-compile, which is the fastest on my AMD Ryzen CPU. SF binaries are taken from abrok.eu (except the official SF-release versions, which are taken form the official Stockfish website).

ORDO calculation fixed to reference-engine (Elo = 0)

You can download all played games from my Google-Drive. Mention: I deleted all comments in the pgn-files (eval, search depth etc.), because the files would be too big otherwise. Download here

Here the progress in regression-testing since Stockfish 14.1 (2021/10/28), with Elo of SF 14.1 set to 0 in a diagram:

     Program                      Elo      +      -   Games   Score   Av.Op.  Draws

   1 Stockfish 220108 avx2    :   40.1    3.5    3.5 20000    55.7 %      0   51.4 %
   2 Stockfish 211227 avx2    :   36.1    3.2    3.2 20000    55.1 %      0   52.1 %
   3 Stockfish 211219 avx2    :   31.5    3.4    3.4 20000    54.5 %      0   51.9 %
   4 Stockfish 211213 avx2    :   29.5    3.3    3.3 20000    54.2 %      0   52.4 %
   5 Stockfish 211209 avx2    :   29.5    3.4    3.4 20000    54.2 %      0   52.5 %
   6 Stockfish 211207 avx2    :   25.3    3.3    3.3 20000    53.6 %      0   53.1 %
   7 Stockfish 211129 avx2    :   23.0    3.2    3.2 20000    53.3 %      0   52.9 %
   8 Stockfish 211127 avx2    :   19.8    3.4    3.4 20000    52.8 %      0   53.2 %
   9 Stockfish 211111 avx2    :    9.0    3.3    3.3 20000    51.3 %      0   54.1 %
  10 Stockfish 211105 avx2    :    8.7    3.3    3.3 20000    51.2 %      0   54.8 %
  11 Stockfish 211121 avx2    :    8.0    3.6    3.6 20000    51.1 %      0   53.9 %
  12 Stockfish 211101 avx2    :    4.1    3.3    3.3 20000    50.6 %      0   54.6 %
  13 Stockfish 14.1 211028    :    0.0    0.8    0.8 340000   52.4 %    -20   50.9 %
  14 Stockfish 210822 avx2    :  -17.5    3.3    3.3 20000    47.5 %      0   54.0 %
  15 Stockfish 14 210702      :  -53.4    3.3    3.3 20000    42.4 %      0   51.4 %
  16 Stockfish 13 210218      : -109.3    3.2    3.2 20000    34.9 %      0   46.9 %
  17 Stockfish 12 200902      : -158.5    3.6    3.6 20000    28.8 %      0   44.8 %
  18 Stockfish final HCE      : -263.4    4.2    4.2 20000    18.2 %      0   30.5 %

Stockfish 210822 was the last dev-version before the Stockfish-Framework started using my Unbalanced Human Openings (UHO) for further Stockfish development...(before that, classical balanced openings were used). Because using UHO openings for development was an important change, IMHO it makes sense to measure the progress from this point and for that, the performance of Stockfish 210822 is needed.

Stockfish final HCE (date 200731) was the latest SF dev-version, before the nnue-neural-nets were introduced. So, this engine is (and perhaps will stay forever?) the strongest HCE (Hand Crafted Eval) engine on the planet. IMHO this makes it very interesting for comparsion.

Below the regression-gamebase recalculated with my Gamepairs Rescorer Batch-Tool. Realizing Vondele's (Stockfish maintainer) idea: "Thinking uniquely in game pairs makes sense with the biased openings used these days. While pentanomial makes sense it is a bit complicated so we could simplify and score game pairs only (not games) as W-L-D (a traditional  score of 2-0, or 1.5-0.5 is just a W)."

   # PLAYER                   :  RATING  ERROR  PLAYED      W      D      L   (%)
   1 Stockfish 220108 avx2    :      78      5   10000   3620   4952   1428  61.0
   2 Stockfish 211227 avx2    :      70      5   10000   3471   5039   1490  59.9
   3 Stockfish 211219 avx2    :      61      5   10000   3325   5086   1589  58.7
   4 Stockfish 211213 avx2    :      59      5   10000   3274   5115   1611  58.3
   5 Stockfish 211209 avx2    :      59      5   10000   3256   5152   1592  58.3
   6 Stockfish 211207 avx2    :      51      5   10000   3159   5115   1726  57.2
   7 Stockfish 211129 avx2    :      46      5   10000   3047   5216   1737  56.5
   8 Stockfish 211127 avx2    :      40      5   10000   2969   5191   1840  55.6
   9 Stockfish 211111 avx2    :      17      5   10000   2575   5346   2079  52.5
  10 Stockfish 211105 avx2    :      17      5   10000   2554   5372   2074  52.4
  11 Stockfish 211121 avx2    :      16      5   10000   2592   5276   2132  52.3
  12 Stockfish 211101 avx2    :       7      5   10000   2400   5411   2189  51.1
  13 Stockfish 14.1 211028    :       0
  14 Stockfish 210822 avx2    :     -34      5   10000   1830   5359   2811  45.1
  15 Stockfish 14 210702      :    -103      5   10000   1208   4729   4063  35.7
  16 Stockfish 13 210218      :    -216      6   10000    503   3502   5995  22.5
  17 Stockfish 12 200902      :    -343      7   10000    177   2125   7698  12.4
  18 Stockfish final HCE      :    -542     11   10000     24    822   9154   4.3

You can download my Gamepairs Rescorer Tool right here