Lc0 or other GPU-Neural Nets versus Stockfish 15.1 testing
Playing conditions:
Hardware: Ryzen 7 6800H 2.6GHz Notebook, RTX 3060 GPU, Windows 11 64bit, 32GB RAM
Cuda version installed: Cuda 12.7
Speed: Stockfish 15.1 plays with 14 Threads (=7 cores) and reaches 10 MN/s in the middlegame. Lc0 minibatchsize parameter is set to the best value for each netsize, depending on Lc0's benchmark with backendbench --clippy.
Hash: 2 GB Hash for Stockfish 15.1 / (NNCacheSize 1000000 or 8192 RamLimitMb for Lc0)
GUI: Cutechess-cli (GUI ends game, when a 5-piece endgame is on the board)
Tablebases: None for engines, 5 Syzygy for cutechess-cli
Openings: UHO_2022_6mvs_+120_+129.pgn. Download my UHO 2022 openings here
Ponder, Large Memory Pages & learning: Off
Thinking time: 2min+2sec for Lc0 and 1min+1sec for Stockfish 15.1: I measured nps on my system and compared these values with the TCEC: My CPU is way too fast, compared with Lc0 running on my RTX 3060 GPU, so it makes sense to set the thinking-time of Stockfish to only 50% of the thinking-time of Lc0. For compensating the fast CPU and the fact, that in TCEC Lc0 benefits from fast hardware and long thinking-time (both is better for Lc0, not for Stockfish)
One testrun takes around nearly 5 days. Average game-duration: 6min 45sec
Each Lc0 / Neural Net plays 1000 games vs. Stockfish 15.1 with my UHO 2022 openings
Learn more about Lc0 (getting started in a GUI, links to net-downloads, FAQs, development-informations and the Leela-Blog) here
Latest update: 2025/01/12: Ceres 1.01 C1-512-35 (with TensorRT installed)
Download all played games (games of the old test-setups, too): here
Program Celo + - Games Score Av.Op. Draws
1 Stockfish 15.1 avx2 : 0 3 3 35000 56.2% -44 49.6%
2 Lc0 0.32dev BT4-6077500 : -11 16 16 1000 48.4% 0 50.2%
3 Lc0 0.31.1 BT4-1740 : -13 15 15 1000 48.1% 0 50.1%
4 Lc0 0.32dev BT4-100 : -13 16 16 1000 48.1% 0 48.3%
5 Lc0 0.31dev T3-2815 : -14 15 15 1000 48.0% 0 47.7%
6 Lc0 0.31dev BT4-6077500 : -14 15 15 1000 48.0% 0 48.7%
7 Lc0 0.31.1 BT4-1130 : -17 15 15 1000 47.5% 0 50.5%
8 Lc0 0.31dev BT4-6315000 : -20 16 16 1000 47.1% 0 51.4%
9 Lc0 0.31dev BT4-5757500 : -21 16 16 1000 47.0% 0 52.1%
10 Lc0 0.31dev TCEC 25 SuFi : -21 15 15 1000 47.0% 0 49.8%
11 Lc0 0.31dev TCEC 25 : -22 15 15 1000 46.9% 0 52.3%
12 Lc0 0.31.2 TCEC 27 : -24 15 15 1000 46.6% 0 49.1%
13 Lc0 0.31dev BT4-6147500 : -25 16 16 1000 46.5% 0 49.7%
14 Lc0 0.31.1 61475pt332 : -25 16 16 1000 46.4% 0 52.6%
15 Lc0 0.31dev 819344 : -31 16 16 1000 45.5% 0 49.5%
16 Lc0 0.31dev BT4-5000 : -33 15 15 1000 45.3% 0 49.4%
17 Lc0 0.31dev BT3-2860 : -35 16 16 1000 45.0% 0 50.6%
18 Lc0 0.31dev BT4-3400 : -37 15 15 1000 44.8% 0 49.6%
19 Lc0 0.31dev 817477 : -38 15 15 1000 44.6% 0 48.2%
20 Lc0 0.30dev T1-4000 : -39 14 14 1000 44.5% 0 49.8%
21 Lc0 0.31dev 817886 : -39 15 15 1000 44.4% 0 50.4%
22 Lc0 0.30dev 811107 : -41 16 16 1000 44.1% 0 46.1%
23 Lc0 0.30dev TCEC 24 : -42 15 15 1000 44.1% 0 51.0%
24 Lc0 0.30rc1 T1-4000 : -44 14 14 1000 43.7% 0 49.8%
25 Lc0 0.30dev T1-30875 : -45 16 16 1000 43.5% 0 47.5%
26 Lc0 0.30dev BT2-4510 : -45 15 15 1000 43.5% 0 47.5%
27 Ceres 1.01 C1-512-35 : -54 16 16 1000 42.3% 0 48.2%
28 Ceres 1.01 C1-640-25 : -61 16 16 1000 41.4% 0 50.6%
29 Ceres 1.01 C1-640-34 : -62 14 14 1000 41.3% 0 49.8%
30 Lc0 0.30.0 815863 : -73 16 16 1000 39.8% 0 47.8%
31 Lc0 0.30rc2 814174 : -80 15 15 1000 38.8% 0 51.0%
32 Lc0 0.30dev 813207 : -84 16 16 1000 38.3% 0 49.6%
33 Lc0 0.30dev TCEC 20 : -90 16 16 1000 37.5% 0 50.5%
34 Lc0 0.30dev T1-2432500 : -94 16 16 1000 36.9% 0 47.2%
35 Lc0 0.30dev TCEC 22 : -95 16 16 1000 36.8% 0 49.4%
36 Lc0 0.30dev TCEC 18 : -133 15 15 1000 31.9% 0 50.5%
Games : 35000 (finished)
White Wins : 17505 (50.0%)
Black Wins : 130 (0.4%)
Draws : 17365 (49.6%)
# PLAYER : Celo Error Pairs W D L (%) CFS(%)
1 Stockfish 15.1 avx2 : 0 ---- 17500 6472 8870 2158 62.3 98
2 Lc0 0.32dev BT4-6077500 : -22 21 500 95 279 126 46.9 55
3 Lc0 0.32dev BT4-100 : -24 21 500 92 282 126 46.6 57
4 Lc0 0.31.1 BT4-1740 : -27 22 500 99 264 137 46.2 52
5 Lc0 0.31dev T3-2815 : -27 23 500 89 283 128 46.1 57
6 Lc0 0.31dev BT4-6077500 : -30 22 500 95 267 138 45.7 61
7 Lc0 0.31.1 BT4-1130 : -34 21 500 85 281 134 45.1 61
8 Lc0 0.31dev BT4-6315000 : -39 22 500 80 285 135 44.5 59
9 Lc0 0.31dev BT4-5757500 : -42 22 500 83 274 143 44.0 54
10 Lc0 0.31dev TCEC 25 SuFi : -44 21 500 78 282 140 43.8 52
11 Lc0 0.31dev TCEC 25 : -44 21 500 85 267 148 43.7 59
12 Lc0 0.31.2 61475pt332 : -48 22 500 80 272 148 43.2 53
13 Lc0 0.31dev BT4-6147500 : -49 23 500 80 270 150 43.0 54
14 Lc0 0.31.1 61475pt332 : -51 20 500 84 260 156 42.8 81
15 Lc0 0.31dev 819344 : -64 21 500 64 282 154 41.0 56
16 Lc0 0.31dev BT4-5000 : -66 23 500 66 275 159 40.7 61
17 Lc0 0.31dev BT3-2860 : -70 22 500 72 257 171 40.1 61
18 Lc0 0.31dev BT4-3400 : -75 22 500 64 267 169 39.5 57
19 Lc0 0.31dev 817477 : -78 22 500 61 269 170 39.1 54
20 Lc0 0.30dev T1-4000 : -79 23 500 62 265 173 38.9 57
21 Lc0 0.31dev 817886 : -82 21 500 53 279 168 38.5 52
22 Lc0 0.30dev 811107 : -83 22 500 53 278 169 38.4 61
23 Lc0 0.30dev TCEC 24 : -87 21 500 56 266 178 37.8 58
24 Lc0 0.30rc1 T1-4000 : -90 22 500 62 250 188 37.4 55
25 Lc0 0.30dev T1-30875 : -93 22 500 60 251 189 37.1 54
26 Lc0 0.30dev BT2-4510 : -94 23 500 60 249 191 36.9 83
27 Ceres 1.01 C1-512-35 : -109 23 500 48 253 199 34.9 84
28 Ceres 1.01 C1-640-25 : -126 23 500 44 239 217 32.7 50
29 Ceres 1.01 C1-640-34 : -126 24 500 43 241 216 32.7 93
30 Lc0 0.30.0 815863 : -151 23 500 34 229 237 29.7 84
31 Lc0 0.30rc2 814174 : -168 24 500 28 221 251 27.7 68
32 Lc0 0.30dev 813207 : -176 25 500 21 226 253 26.8 78
33 Lc0 0.30dev TCEC 20 : -190 26 500 25 203 272 25.3 74
34 Lc0 0.30dev T1-2432500 : -202 26 500 20 200 280 24.0 58
35 Lc0 0.30dev TCEC 22 : -206 26 500 25 186 289 23.6 100
36 Lc0 0.30dev TCEC 18 : -315 32 500 12 118 370 14.2 ---
-------------------------------------------------------------------
--- Number of all Gamepairs : 17500
--- Number of drawn Gamepairs overall: 8870 (= 50.68%)
--- Number of 1:1 drawn Gamepairs : 4470 (= 25.54%)
--- Number of 2-draws drawn Gamepairs: 4400 (= 25.14%)
-------------------------------------------------------------------