29 April 2012

Pitfalls of Computer Analysis


After having worked through and published over forty annotated games, in the process employing what I'd consider to be a simple and straightforward method of computer-assisted analysis, I've acquired an idea of some of the major pitfalls, as well as the benefits, of using chess engines.  In sharing the following observations, it's worth noting that they are offered from the point of view of an improving Class player, rather than that of a professional chessplayer or computer programmer, whose needs and approaches to using computers may well be different.
  • Ironically in my view, one of the most pernicious pitfalls for the improving player is psychological: the sense of failure that can result after seeing computer analysis repeatedly demonstrate wins missed or ways that a loss could have been avoided.  How often do you see players comment about how stupid they feel or how bad their chess is following a game?  It's important to have the mental toughness to stop kicking yourself afterwards and truly learn from your mistakes, rather than waste mental energy on self-denigration.  Let the chess engine serve its function as a constructive tool to be used for improvement purposes.
  • While outright blunders and forced wins are easily highlighted by engine analysis, the "best move" in a position is often not so easy to determine.  This may even be impossible, given broadly conflicting plans and ideas: see what super-GM Kramnik has to say about this.  In practical terms, engine evaluations that differ by 0.1 pawns are essentially equivalent and I would not sweat too much over a difference of up to 0.2 pawns.
  • Only looking at a single visible line of analysis (the engine's top choice) can therefore be misleading when trying to understand a position; unfortunately, that is usually the default setting for a lot of software programs.  For analysis purposes, a player needs to have multiple move choices (lines) visible in order to better comprehend the potential of a position; having 3 lines visible is my preference.
  • Trusting the initially displayed engine move evaluations can be counterproductive to analysis and misleading.  Engines need time to sort through possibilities in non-forcing situations and at least 1-2 minutes should be allowed when looking at a move using "infinite analysis" before proceeding.  Stepping through subsequent moves of key variations is also very important and may change the evaluation of the line, after the engine is able to calculate further ahead in it.
  • Computers are of less practical help in objectively evaluating best opening play, due to the sheer number of possibilities and lack of forcing tactics leading to an advantage.  The best use of an engine in the opening phase is instead to search for and evaluate alternative moves that don't appear in, or are neglected by, opening theory; this is how the professionals use them to find theoretical novelties.
  • Playing a move suggested by an engine is dangerous when you don't understand why it should be played.  This is one of the main reasons that blindly following engine recommendations in the opening can be counterproductive, even if they are objectively "best" from the engine's point of view.  Ending up in a middlegame position that you don't know how to play is often a quick route to defeat.
  • In situations without forcing tactics, different engines may have significantly different assessments of the position.  It's therefore important to understand as best you can the particular biases and limitations of the engines you use, and/or use multiple ones for analysis.  Common biases include over-valuing material and mis-evaluating endgames in the absence of tablebases.  For example, having initially run through my past tournament games with Fritz and now using Houdini to assist in analyzing them, I've run across several instances of key positions where Fritz preferred a different move, while Houdini has supported my original evaluation and move choice.
In the end, the point of seriously analyzing your games (or anyone else's) is to achieve a greater understanding of them and through that, of the game of chess in general.  It's therefore wise to remember that the computer is there to support and asisst you, not to function as some sort of ultimate arbiter of what is correct and should always be played.

7 comments:

  1. Great post!

    1) I think having a few engines helps to get different ideas. "Strength" of the engine isn't super important as they are all strong. For instance an engine I LOVE is Smarthink 1.20! On the engine lists it is rated at about 2800 or so which of course is Grandmaster strength but low compared to Rybka, Houdini et. al. BUT Smarthink VERY OFTEN comes up with some interesting moves that the other engines didn't consider. Can be very educational when doing a post game analysis!

    2) I always go through the game on my own first and then with the engine. (Houdini as my first engine) But when i go through the game with the engine I set the GUI so that I cannot see the engine's move choice, only it's evaluation. So whenever the evaluation changes enough to warrant a look I THEN try to pick a better move! After I do that I will then and only then check to see what the engine's move choice is. So each post game analysis turns into a bunch of mini-styoko exercises.

    ReplyDelete
  2. Dang Tommy that's a great idea in how to learn from an engine w/ the stoyko exercises. Really good post you made here Admin. Personally I don't pay attention to any of the engine's suggestions to a move I could've played unless there's a difference in 3/4th of a pawn. Anything other then that is pretty negligible to me against players of my caliber. Also, I think using engines too often to "vindicate" that your position WAS actually won promotes pigheadedness and retards learning. Only now and again did I bother using my silicon friend.

    ReplyDelete
  3. I don't use it as much as I used to, but one thing I always liked was that it showed my missed "removal of the guard" tactics pretty clearly. Those patterns aren't always immediately visible (unlike pins and such), so I thought it was useful to try to pound those into my feeble brain.

    Funny that you say 3 top lines - that's what I always set mine to. Much more useful than having just one - lots of times my move didn't agree with Shredder's, but it was within 0.02 or something when I expanded to add the extras.

    One other thing I think it can be good for is the technique for converting a (slightly) won ending. I always go back when I draw/lose one I thought I was ahead enough to win to see where I screwed it up and the technique for next time.

    ReplyDelete
  4. Thanks for the comments, I think this is one of those topics where people have varied, useful experiences.

    I always like to hear about what engines are out there and their utility, Smarthink is new to me. I found out about Houdini originally (before it was marketed in a retail form) via a brief mention in a USCF online article.

    It's worth noting that some top GMs use the technique of viewing engine evaluations of a position, but not displaying the engine's move, when doing their own analysis; I've seen at least Kramnik mentioned as doing that.

    When I get to seriously studying endings, I'm definitely going to be practicing winning won positions vs. the computer, and perhaps vice versa to see the technique demonstrated.

    One of the things I didn't bring up in the original post was the increasing meaninglessness of the numerical evaluation after a certain point in the game. Once you have a won game, it's not really that significant if your position is rated as 2.5 pawns ahead or 4.5 pawns - unless of course you fritter or blunder away the advantage. There is a big difference between a 0 (drawn) evaluation and a +/- 0.75 one, though.

    ReplyDelete
  5. A good post, with it and the comments providing food for thought.

    I particularly agree with the "mental toughness" and self-deprecation thoughts. There are many times when using engines that my first thought has been how stupid I am not to see something, but you are right, this isn't the correct attitude.

    Engines easily see tactics ( simple and complex ) and provide a harsh tutor that them.

    However, where tactics aren't immediately involved, if a human player ( er, that's me :) actually has a plan and a reason for playing what is not considered to be the best move, then a lot can be learned in viewing both the straight engine refutation, and using it to investigate 'what ifs' against your plan.

    If you have no plan when you play, of course, its a different matter.

    I am not sure I can only view the evaluation in Chess Assistant, but will have a look as that is a great idea.

    As for engines, Houdini ( the free 1.5 ) is my poison, with an old shredder (9.11) and the interesting Aransan providing completely different views and assessments to the Super-GM silicon monsters. Rybka ( despite all the accusations ) is still damned good and more than a match for Houdini, so I activate the beast occasionally.

    The other option I have is to switch from an absolute number assessment to the symbolic one, ie +/= etc...that way the 0.2 and 0.3 disappear and all you have is the more human-biased guide.

    ReplyDelete
  6. Hey Signalman,

    I like the idea of just switching to the symbolic evaluation! At least for a first run through. Although the numeric evaluation helps when doing my "mini-styoko hidden engine choice" analysis as I can see that (as mentioned above) a move I may choose might not be the exact move the computer picks but would still be in a healthy numeric range.

    For GUIs, I am sold on Shredder Classic from Shredder Chess.com It allows me the most flexibility when doing my analysis.

    And I agree that some of the older engines can supply more interesting moves!

    ReplyDelete
  7. A very good post and interesting topic I thought.

    I don't use computers much but I've found that their "analysis" can be very misleading. Since they often base their candidate choice on the sequence of best moves for both sides going out quite a bit, any interpolation of a good solid move rather than best can change the eventual play sequence from a positive to equal strangely enough.

    While this creates better chances for drawing against a computer it would increase the chances for a win against a human line that can't calculate out that far and might tend to get out of position.

    I saw a recent example in the Aronian-Kramnik match where it looked like a computer based line got busted with a fairly simple defensive move.

    So I guess I very much agree with your view "Playing a move suggested by an engine is dangerous when you don't understand why it should be played."

    ReplyDelete

Your comments and ideas on chess training and this site are welcomed.

Please note that moderation is turned on as an anti-spam measure; your comment will be published as soon as possible, if it is not spam.