Just added to our tools section the ability to validate your robots.txt file. There are tons of robots.txt validators available but this one is slightly different in that it looks at the file from an SEO perspective – looking to see if you’ve blocked too many pages from the spiders, whether there’a sn XML sitemap, etc. It validates not only the standard robots.txt syntax but also more modern additions that are not truely standardized yet, such as the commands ‘sitemap’, and ‘allow’.

Give it a try and let me know if the validator finds any errors in your robots.txt file.… Read the rest