Comparing move choices of chess search engines

Anecdotal evidence exists that in many positions two distinct chess engines will choose different moves and, moreover, that their top-n ranking of move choices also differ. Here we set out to quantify this difference, including the difference between move choices by chess engines and those made by humans. For our analysis we used FRITZ 8 and JUNIOR 9 as representative chess search engines and the POWERBOOK opening book as representing human choices. We collected the top-5 ranked moves and their scores as reported by FRITZ and JUNIOR, after 15 and 30 minutes of thinking time, and the top-5 moves recorded in the POWERBOOK, for the Nunn2 test positions and the initial board position. The data analysis was carried out using several nonparametric measures, including the amount of overlap in the top-5 choices of the engines and their association as measured by three variants of Spearman's footrule. Our preliminary results show that, overall, the engines differ substantially in their choice of moves, and, furthermore, the engines' choices also differ substantially from human choice.