Stefan Pohl Computer Chessprivate website for chessengine-testsNBSC Advanced Armageddon Openings for Computerchess - the future of Computerchess (Part 2)
No Black Short Castling advanced scoring Armageddon openings
No longer any draws in computerchess !!! No boring draw-marathons, no draw-death of computerchess – never!
Download NBSC Advanced Armageddon openings here
Base idea is from Larry Kaufman (Komodo-Team): The first idea for Armageddon openings for computerchess was, to allow white to castle short, only and black to castle long, only. This so called SALC-Armageddon worked pretty well in my testruns. But, some months ago, I had the idea for a new advanced scoring system for Armageddon computerchess:
Classical Armageddon scoring is: Win for white = 1 point for white Draw = 1 point for black Win for black = 1 point for black
My Advanced Armageddon scoring is: Win for white = 1 point for white Draw = 1 point for black Win for black = 2 points for black (!!!)
Because the chessengines dont know, that they are playing Armageddon-Chess, I believe, it makes sense, to use the advanced scoring: The engines try to win a game, even though they play with black and in the classical Armageddon scoring, there is no difference between a win for black and a draw. So, there is no benefit for the engine, to win a game, when playing black. With my advanced Armageddon-scoring, black gets 2 points for winning a game, because the Armageddon-openings give a measureable advantage for white and it is very difficult for black to win a game. So, trying to win a game even though the engine plays black, makes sense with the advanced scoring, because black gets 2 points for a win. With the classical Armageddon scoring, black does not benefit from winning a game. It is clear, that the advanced scoring-system lowers the whitescore-level. And in SALC-Armageddon, the whitescore was not very high even though the classical Armageddon scoring is used. So, I decided to give a white a higher advantage and allow white to castle long and short. Black is still allowed to castle long. This „NBSC“ (=No Black Short Castling) Armageddon gives white more advantage and increases the whitescore. The NBSC line is 1. Na3 Nh6 2. Nb1 Rg8 3. Nc3 Rh8 4. Nb1 Ng8
I built 3 NBSC-Armageddon opening-sets (and some small tools for classical and advanced Armageddon scoring):
NBSC_Noomen_3mvs: Built from the Noomen 3mvs opening set (deleted double endpositions and opening-lines, which are not good for computerchess IMHO (Dutch, King's gambit etc). 250 lines remaining.
NBSC_SuperGM_4mvs: 2612 lines. Both players 2600+ Elo. Out of the Megabase 2018. All endpositions calculated by Komodo 13.1 (7 hyperthreading Threads on a Quadcore, 30''/move). All endpositions in an eval interval of [-0.40/+0.60] (before the NBSC-line was added, of course).
NBSC_30k_5mvs: 30578 lines. Both players 2300+ Elo. Out of the Megabase 2018. All endpositions calculated by Komodo 13.1 (7 hyperthreading Threads on a Quadcore, 20''/move). All endpositions in an eval interval of [-0.40/+0.60] (before the NBSC-line was added, of course).
I recommend to use the smallest of the 3 opening sets, which is big enough for your engine play or testing.
The Book-folder contains openings-books for FritzGUI, ArenaGUI and polylgot-book format (the ShredderGUI has a serious bug in its opening-book using: It does not recognize, that the 2 move sequence of the rook (from h8 to g8 and back to h8) makes the short castling for black illegal. So, an opening-book built from NBSC-lines will not work in the ShredderGUI)
Here the results of the testruns (the results of NBSC Armageddon openings are just mind-blowing!):
Testing conditions: 2'+1'', Singlethread, i7-8750H Hexacore mobile CPU, 256MB Hash, cutechess-cli (no TB for engines, but 5 Syzygy for cutechess), Contempt=0 for all Stockfish. All openings replayed with reversed colors. Round Robin with 1500 games with SF 11, SF 10, SF 9 and SF 8. Each SF played 250 games vs. Each of the 3 opponents = 1500 games per testrun. ORDO for the ratings (3400 Elo base-value)
8 classical (non-Armageddon) opening-sets for comparsion: Balsa_2724, Chad 8ply, Hert500, Stockfish Framework 8moves v3, SuperGM 4moves, SALC V5 fc 500, Drawkiller tournament and J.Noomens "action" openings (TCEC, Kings Indian and Gambit-lines in one file). Sorted by Elo-spreading (Elo-distance between SF 11 and SF 8 (first to last)).
Balsa 2724: Program Elo + - Games Score Av.Op. Draws Draw-rate : 72.9 % (smaller is better)
Program Elo + - Games Score Av.Op. Draws Draw-rate : 67.5 % (smaller is better)
Program Elo + - Games Score Av.Op. Draws Draw-rate : 67.3 % (smaller is better)
Stockfish Framework 8moves v3: Program Elo + - Games Score Av.Op. Draws Draw-rate : 69.9 % (smaller is better)
Hert 500: Program Elo + - Games Score Av.Op. Draws Draw-rate : 67.4 % (smaller is better)
Program Elo + - Games Score Av.Op. Draws Draw-rate : 63.1 % (smaller is better)
Program Elo + - Games Score Av.Op. Draws Draw-rate : 52.8 % (smaller is better)
Program Elo + - Games Score Av.Op. Draws Draw-rate : 41.9 % (smaller is better)
[XXX]= results with classical Armageddon scoring
NBSC Advanced Armageddon Noomen 3-moves: Program Elo + - Games Score Av.Op. Draws Draw-rate : 0 % (smaller is better)
Program Elo + - Games Score Av.Op. Draws Draw-rate : 0 % (smaller is better)
Program Elo + - Games Score Av.Op. Draws Draw-rate : 0 % (smaller is better)
As you can see, the NBSC-Armageddon results are "out-of-this-world": no draws, of course, and an Elo-spreading, that ist around 2.25x wider, than using "classical" openings: Average Elo-spreading of Balsa, SuperGM, Chad 8ply, SF Framework openings, HERT 500 and J.Noomen openings is = 142 in this test-series. Average Elo-spreading of the NBSC Armageddon openings (3-5 moves) is = 318 (!!!)
To make clear, what this really means, mention, that you need an errorbar in your engine-testings, which is 2.25x bigger, when using classical openings, compared to the errorbar, you need, when using NBSC Advanced Armageddon openings, to make a ranking of engines (Engine A better than B, better than C) statistically valid. Because the ratings of engines are spreaded so much by the NBSC Advanced Armageddon openings. And mention, that you need around 4x more (!) games in order to half the size of the errorbar. That means, you have to play only around 20%-25% amount of games for the same statistical quality of testing-results, when using NBSC Advanced Armageddon openings, compared to classical opening-sets !!! So, all testwork can be done in a 75%-80% shorter time. Or, when you play the same number of games, the statistical quality of the results is way better than ever before!
IMPORTANT rules for using the NBSC Armageddon Openings: Not made for playing no Armageddon - so it is absolutely necessary to set all 1/2-1/2 results (draw) of the played games to 0-1 (win for Black), when an engine-tournament or testrun is finished. Not made for playing versus other books or opening-sets: In an engine-tournament or testrun ALL engines must use the same Armageddon openings-set (or book).
Base idea of Armageddon openings: Larry Kaufman (Komodo-Team) Developed and constructed by Stefan Pohl (SPCC)
(C) NBSC Advanced Armageddon Openings: Stefan Pohl (SPCC) 2020
|