Stefan Pohl Computer Chessprivate website for chessengine-testsLC0 / Neural Nets versus Stockfish testing
Playing conditions:
Hardware: i7-8750H 2.6GHz (Hexacore) Notebook, RTX 2060 GPU, Windows 10 64bit, 16GB RAM Speed: Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off): 9000 kn/s, Lc0 (with old 32930 20x256 net): 16000 n/s in starting position. Since Lc0 0.26.3, Lc0 uses Cuda 11.1, which makes it around +37% faster. To keep the Leela-Ratio on 1.0, I slowed down the GPU with the Afterburner-Tool. Hash / NN Cache: 4096 GB Hash for Stockfish / 5000000 NN-Cachesize for Lc0 GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board) Tablebases: None for engines, 5 Syzygy for cutechess-cli Openings: NBSC Advanced Armageddon Noomen 3-moves (250 openings). Learn more about Advanced Armageddon in the "NBSC Armageddon openings"- section and download the NBSC-Armageddon package right here Ponder, Large Memory Pages & learning: Off Thinking time: Lc0 2'+1'' and Stockfish 3'+1.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 8 minutes, one 500 games-testrun takes around 2.5 days.
Each Lc0 / Neural Net plays 500 games vs. Stockfish with my new NBSC Advanced Armageddon openings. After the testrun is finished, all games are rescored with my armageddonize_advanced-tool. Means: Win for white = 1 point for white
Learn more about my new NBSC Advanced Armageddon openings and the advanced scoring system in the "NBSC Armageddon openings"- section. Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here
Latest update: 2020/01/13: Lc0 0.26.3 66988. Next testrun Lc0 0.26.3 J94-100
(Since Lc0 0.26.3, Lc0 uses Cuda 11.1, which makes it around +37% faster. To keep the Leela-Ratio on 1.0, I slowed down the GPU with the Afterburner-Tool.)
Download all played games (non-armageddonized) here
500 NBSC-Advanced-Armageddon games each testrun (= a win for Black is 2 points for Black and a draw is a 1 point-win for Black). vs. Stockfish 200418 (SPCC-Elo: 3568 (Contempt set to 0) (around +14 Elo stronger than Stockfish 11 (SPCC-Elo: 3554)). The errorbar of each result is +/- 20 Elo. But mention, that the usage of my NBSC-Armageddon openings spreads the Elo-results around 2.25x wider, than using classical openings for testing(!), so with classical openings, you would need an errorbar of +/- 9 Elo for the same statistical quality of the results (= the rankings of Lc0 nets here). And for an errorbar of +/- 9 elo, you need around 3000 games, not 500, which means 6x more games (and 6x more PC-time)!! Learn more about that revolution in computerchess in the "NBSC Armageddon openings"- section of my website.
# PLAYER : RATING ERROR PLAYED W L (%)
White Wins : 18091 (51.6 %)
Mention, that this is not a ratinglist, but only a performance test of Lc0 with different NNs versus Stockfish. Because Lc0 vs. Stockfish is definitly the most interesting head-to-head competition of NN vs. AB-engines. For a real ratinglist including Lc0 running on a RTX-GPU (with a valid Leela-Ratio of 1.0), please visit Andreas Strangmueller's excellent website. Just click here
Stockfish (with nnue) vs Lc0 longtime testing
Each testrun 300 games with 150 Noomen lowdraw-openings (selected openings from TCEC superfinals) and 5'+3'' thinking-time (Lc0) / 7.5'+4.5'' (Stockfish). This thinking-time gives a perfect Leela-Ratio of 1.0 on the used PC hardware: i7-8750H 2.6GHz (Hexacore, TurboBoost mode off) Notebook, RTX 2060 GPU. Average game-duration: 20 minutes. Stockfish (running on 11 hyperthreading-threads, Intel Turbo-Mode off) 9000 kn/s, Lc0 (with old 32930 20x256 net) on RTX 2060 mobile: 16000 n/s in starting position. Hash / NN Cache: 4096 GB Hash for Stockfish / 10000000 NN-Cachesize for Lc0 GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board) Tablebases: None for engines, 5 Syzygy for cutechess-cli Openings: 150 Noomen lowdraws openings (J. Noomen selected non-drawish openings out of his TCEC superfinal openings of previous TCEC seasons). Download here Ponder, Large Memory Pages & learning: Off Thinking time: Lc0 5'+3'' and Stockfish 7.5'+4.5'' (means a perfect Leela-Ratio of 1.0). Average game-duration: 20 minutes.
Download all played games here
Latest update: 2020/12/30 Stockfish 201225 vs Lc0 0.26.3 66680 See some short and spectacular wins of this match directly here on the website in the "View SF vs Lc0 games"- section! Stockfish 201225 bmi2 vs Lc0 0.26.3 66680 : 300 (+ 60,=223,- 17), 57.2 % (+50 Elo)
|