HTML encoding errors will break search engine spiders. To understand why you need to think of the differences between browsers and spiders:

Browsers have been written to be backwardly compatible with widely used, badly formed HTML. This is due to user insistence that sites that work in one browser also work in another, no matter how buggy the code: if they didn’t then the user would swap to a browser that did ‘work’. Spiders do not have this requirement, if your site can’t be read then there will be others that can be.

Browsers have kept pace with the internet and have been developed and enhanced through the years. Spiders still behave as Mozilla 2.0 did 7 years ago.

Don’t be fooled by other sites out there that have HTML errors. Not all HTML errors will cause spiders to break, but some will.

Not all sites need to be spidered – Dell, Gateway, IBM, Nissan, Ford, Honda, all these companies have spent a lot of money putting their brand into your mind as the place to go for computers, cars, etc. People who want information on a Ford will go to www.ford.com without needing to go via a search engine.

Here’s an article on this issue: http://www.netmechanic.com/news/vol5/promo_no23.htm