Another bug hunting session

As mentioned in an earlier post I develop the ice engine with MSVC and compile the release versions with the gnu c++ compiler as it creates the faster executable. At some point in time I realized that both executables compiled from the same source produced different outcomes.

When searching for a move suddenly the number of processed nodes was different, different pvs were calculated and sometimes also different best moves were returned. So I knew there was a bug some where but it was very hard to spot as both programs just for themselves appeared to run fine.

I reversed recent code changes to spot the source of the error but it did not help, both versions behaved differently, so I started the bug hunting session. Errors produced by the search algorithm are very hard to spot because the algorithm is recursive and calls itself several thousand times a second. I finally was able to spot the last position were both programs were equal, so I knew this position was somehow the cause of the problem. It showed that eval returned in both executables different scores for that position. From now on it was easy as eval is not recursive.

The root cause was an array with pawn bonuses per rank. The array was sized too small. In some positions the access to that array just exceeded the bounds of the array and retrieved a random number that just happened to be at the memory position. This random number was not the same in the MSVC and the gnu version and so the execution started to run differently.

This was then a quick fix and now both versions are consistent again.

I ran a quick tournament (50 games) of the latest ice version against a former not too old release to measure whether recent changes improved the strength of the engine. And the latest release scored

38 : 12

So although I consider the engine rather weak at least there seems to be some progress.

Chess engine input handling

When designing the ice engine I decided to design it as a single threaded program to save the thread handling overhead.

In the pascal mace engine I use threads for the input handling. I read from stdin, when no input is available the execution stops. This is unfortunate if this is your main thread, it is no problem when it is an extra thread created just for that purpose. In ice I wanted to use PEEK_NAMED_PIPE to look whether input is available and only start reading if it signals that input is there, so the execution is not blocked by reading from an empty input device.

It worked most of the time, it failed when I gave the console window, where the engine is running in, focus or removed focus. Although I specified that I don't want to have window events PEEK_NAMED_PIPE returned true in those cases, it tried then to read from stdin and execution stopped as no real input was available.

This only happens when I run the engine in a console window not when the engine is running as a process started by the GUI (try to give focus to a process ...) but it is annoying anyway.

I returned to the thread method for handling stdin input. I use the standard methods from Windows to do that and the performance hit I'm taking seems minimal.

Next to last mistake

Victory goes to the player who makes the next-to-last mistake.
Savielly Grigorievitch Tartakower, Chess Master

While working on the engine evaluation I thought it might be helpful for me to play some games against other real players to get some fresh ideas. It's been a while since I did that.

I went to yahoo chess and played a few games. As a noob I was given a low provisional score and engaged games with players that had a comparable score. To my surprise there were players with a low score like me that played already thousands sometimes more than ten thousands games. I don't think I played ten thousand games in my whole life and if I had I would probably have a better score.

Anyway the last game was a real challenge for me. My opponent played very aggressive and I was somehow intimidated by that. I thought I was screwed but I just wasn't able to find the best moves. I performed an analysis of the game after wards with the help of my engine and it showed that we both played terrible. His gambit offers were incorrect so was my declining, I overlooked easy tactical combination's.

Here is the game (I'm playing White)

1.  e2-e4 e7-e5
2.  g1-f3 b8-c6
3.  f1-c4 g8-f6
4.  d2-d3 f8-c5
5.  o-o   f6-g4
6.  d1-e2 d7-d6
7.  h2-h3 h7-h5

I thought it is too dangerous to accept the gambit and take the knight, because of the open h-file with blacks rook on it. But after 9. Ng5 it would probably be ok to take it. Instead
 

8.  b1-c3 c8-e6
9.  c3-d5 c6-d4
10. e2-d2 d4xf3+
11. g2xf3 d8-h4
12. d2-g5 

Black is too greedy

12. ... h4xh3
13. g5-e7++

At the end he was so occupied with his assault that he overlooked the Mate in 1 threat and this one time I was aware of it, so I won luckily. But I take this a s a warning, I need more discipline. Calculate better and consider all alternatives. Play a bit like my engine plays.

 

  

Evaluation evaluation

Last time I thought about the evaluation part of the engine. This part is responsible for returning an absolute score for a certain position. Here all the chess specific knowledge is contained. The engine must be able to see whether a certain position is in favor for white or black, so it knows which position it should avoid and which positions it should play towards to.

Easy evaluation terms are material based. The side with more material has an advantage. The bigger the material difference the higher the advantage. But even material becomes difficult when you try to fine tune that.

Example question: What is better, a queen or two rooks ?

The answer depends on the other material on the board, if there are a lot of pawns on the board, the queen might be superior because the rook paths are blocked. If there are only a few paws on the board the rooks gain a lot of strength.

A queen that is not opposed by a bishop pair is stronger than one were the opponent still possess the bishop pair.

So to tune the evaluation is probably the must difficult part and also the part the engine gets its personality from. What focus I give on certain terms will determine the engine play, so it is quite an interesting subject.

For the programming I maintain a header file with all the evaluation terms and scores associated. This is very time consuming to maintain especially if you want to have a nice code formatting.

I started to maintain the evaluation terms in a Excel and wrote a Visual Basic macro that exports the terms into c++ code. This is really helpful.

.

Including binary chess endgame tables into ice executable

In order the include basic functionality into the new c++ based ice chess engine I moved to basic endgame play. Basic endgame means that the engine should understand when a 3 man endgame is a draw, when it is a win and how to play it correctly. The most accurate way is to make endgame tables for KPK, KRK and KQK available to the engine.

From my former work with endgame table for the old mace engine I calculated all those tables and it was time to use them for ice also.

Because this knowledge is so essential and the engine relies on having it available I decided to include the tables into the binary executable and not to distribute them as separate files along with the engine.

To do that I added a function to my table base generator that generates c++ header files that include the contents of the table.The generated file looked like this.

#ifndef KPK
#define KPK

/**************************************************************
| Chess Endgame Table Base Data for Endgame Type : KPK        |
|-------------------------------------------------------------|
| created by table base creator 1.0                           |
| (c) by Thomas Petzke                                        |
| 23.01.2011 18:18                                            |
|**************************************************************/

const int KPK_TBL_SIZE = 79865;
const short KPK_TBL[KPK_TBL_SIZE] = {
-4080,
-4080,
-3968,
-3968,
-3825,
-4080,
-4080,
-4080,

...

I estimated that the executable growths by a few 100k in size when this tables are included. To my surprise the gnu c compiler added almost 2MB to the size of the executable, so much more than necessary for the raw data. Obviously he aligns the array content for faster access so each element takes 8 bytes (instead of the 2 bytes of a short).

As speed is not really a problem here, the array is read only once when the program initializes, so when this takes a few ms longer no one cares, I tried to shrink the size of the executable.

I packed 4 shorts into an 64bit integer, but the size remained the same. Also using pre processor directives did not help. So I minimized the array size by not including illegal positions and easy to recognize draws in it. This helped a bit but the executable is still larger than it should be.

As it is more a cosmetic problem I decided to live with it for the moment. At least it works correctly :-)

White moves and mates in 23 

iCE 0.1 build 511 [2011.1.24]
position fen 8/8/8/8/8/6k1/4P3/2K5 w - - 0 1
go depth 1
info depth 1 seldepth 0 time 0 nodes 7 pv c1c2  nps 6999 score mate 23 hashfull 0 tbhits 7
bestmove c1c2 

Solving the FINE70 position

So after fixing the last bug (see last post) the engine solves to FINE70 position correctly again. Here is the engine protocol about it for reference

iCE 0.1 build 333 [2011.1.6]
fine70
go 26 ply at FINE70 = 8/k7/3p4/p2P1p2/P2P1P2/8/8/K7 w - -
info depth 1 seldepth 1 time 0 nodes 6 pv a1b1  nps 5999 score cp 97
info depth 2 seldepth 2 time 0 nodes 17 pv a1b1 a7b6  nps 16999 score cp 97
info depth 3 seldepth 3 time 0 nodes 68 pv a1b2 a7b6 b2c3  nps 67999 score cp 99
info depth 4 seldepth 4 time 0 nodes 95 pv a1b2 a7b6 b2c3 b6c7  nps 94999 score cp 99
info depth 5 seldepth 5 time 0 nodes 178 pv a1b2 a7b6 b2c3 b6c7 c3c4  nps 177999 score cp 99
info depth 6 seldepth 6 time 0 nodes 216 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7d7  nps 215999 score cp 99
info depth 7 seldepth 7 time 0 nodes 310 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7d7 c4d3 nps 309999 score cp 99
info depth 8 seldepth 9 time 0 nodes 413 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 b6c7 nps 412999 score cp 99
info depth 9 seldepth 9 time 0 nodes 477 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 b6c7 d3c4  nps 476999 score cp 99
info depth 10 seldepth 11 time 0 nodes 907 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 906999 score cp 99
info depth 11 seldepth 11 time 0 nodes 764 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 763999 score cp 99
info depth 12 seldepth 13 time 0 nodes 1252 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 1251999 score cp 99
info depth 13 seldepth 13 time 0 nodes 1337 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 1336999 score cp 99
info depth 14 seldepth 15 time 16 nodes 1790 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 111874 score cp 99
info depth 15 seldepth 16 time 0 nodes 1771 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 1770999 score cp 99
info depth 16 seldepth 18 time 15 nodes 2284 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 152266 score cp 99
info depth 17 seldepth 18 time 16 nodes 2582 pv a1b2 a7b6 b2c3 b6c7 c3c4 c7b6 c4d3 nps 161374 score cp 99
info depth 18 seldepth 21 time 62 nodes 20832 pv a1b2 a7a8 b2c3 a8b7 c3c4 b7b6 c4d3 nps 336000 score cp 99
info depth 19 seldepth 20 time 0 nodes 2508 pv a1b2 a7a8 b2c3 a8b7 c3c4 b7b6 c4d3 nps 2507999 score cp 99
info depth 20 seldepth 22 time 0 nodes 3554 pv a1b2 a7a8 b2c3 a8b7 c3c4 b7b6 c4d3 nps 3553999 score cp 99
info depth 21 seldepth 24 time 16 nodes 4393 pv a1b2 a7a8 b2c3 a8b7 c3c4 b7b6 c4d3 nps 274562 score cp 99
info depth 22 seldepth 25 time 31 nodes 5572 pv a1b2 a7a8 b2c3 a8b7 c3c4 b7b6 c4d3 nps 179741 score cp 99
info depth 23 seldepth 26 time 0 nodes 6317 pv a1b2 a7a8 b2c3 a8b7 c3c4 b7b6 c4d3 b6c7 nps 6316999 score cp 99
info depth 24 seldepth 28 time 16 nodes 7638 pv a1b2 a7a8 b2c3 a8b7 c3c4 b7b6 c4d3 b6c7 nps 477374 score cp 99
info depth 25 seldepth 30 time 78 nodes 72402 pv a1b1 a7b7 b1c1 b7c8 c1d2 c8d7 d2c3 d7c7 nps 928230 score cp 199
info depth 26 seldepth 31 time 63 nodes 45329 pv a1b1 a7b7 b1c1 b7c8 c1d2 c8d7 d2c3 d7c7 nps 719507 score cp 199
bestmove a1b1

There is always one more bug

Today I ran into an extremely annoying bug. I found it when I was doing a stress test against my transposition table. I do that once in a while to see that recent changes did not break anything.

Usually I run a ply 26 search against the position known as FINE70 (or the Lasker-Reichhelm Position)

8/k7/3p4/p2P1p2/P2P1P2/8/8/K7 w - -

The only winning moves is Kb1!. All other moves lead to draw. With a correct implemented transposition table the position is solved in about 1 sec or less, without a transposition table or with errors in it it takes much longer or is not solved at all.

The last test revealed that the ice engine was not able to solve it anymore. It ran quite fast through the plys but at ply 26 it showed Kb2 as best move.

So the fun began, trying to find an error in the transposition table. I found several bugs where I used & in a logical expression where it should read && but this did not solve the problem.

The last thing I added was considering piece square values in the evaluation and indeed when I removed that code the position was solved. But the code is really simple, I could not imagine I did anything totally wrong here, I debugged it anyway but to no result. It worked as it should just when it added a small value to the score less than 1/10 of a pawn the position was not solved anymore.

At the end I turned the hash table off and let the engine run without it (it calculated over an hour) and to my surprise the position wasn't solved either. So obviously it was not a hash table problem. So I went to the PVS search, turned off extensions, quiescence search, LMR, zero window search in PVS, everything I could think off. Trying to debug a 26 ply search in a recursive algorithm is really mind boggeling. I wasn't able to spot any error, with every feature alone or all together turned off, the position was still not solved.

I finally started to collect the principle variations at ply 25 to see what happens to the winning move line and to my surprise at ply 25 no principle variation with an exact score that contained the winning move Kb1 was encountered. It seemed that this move and so no line starting with it was existing in the search tree, which was quite odd. But this revealed the most likely source of the error to me. It had to be in the code where the root position is handled. Here for every legal move in the root position a PVS search is performed. (The root node is outside the normal search, because at the root I need more than the score of the position, here I need the best move also).

It showed that I did not run the search for all moves at the root, I skipped the last one, the code was

for (i=0; i < ml->moveCnt -1 ; i++)

It did not show as a problem before because the move list is usually well ordered and the last move will very unlikely turn out as the best. Adding the piece square evaluation to the the score changed the order for the moves in the list, Kb2 was considered better than Kb1 on lower plys (king closer in the center), this changed the order of the moves in the root position and Kb1 was moved to the end for searches from ply 4 onwards. Here it was not considered anymore because of the -1 bug.

So my head hurts from recent head banging but I'm glad I found that nasty little bug. It really bothers me knowing to have a bug somewhere in the code and no be able to find it.

So this one is fixed now, but I already encountered the next one. There seems to be a problem with the zero window search in PVS. There is always one more bug ...