Robots.txt validator

March 11, 2009 – 2:55 pm

Just added to our tools section the ability to validate your robots.txt file. There are tons of robots.txt validators available but this one is slightly different in that it looks at the file from an SEO perspective – looking to see if you’ve blocked too many pages from the spiders, whether there’a sn XML sitemap, etc. It validates not only the standard robots.txt syntax but also more modern additions that are not truely standardized yet, such as the commands ’sitemap’, and ‘allow’.

Give it a try and let me know if the validator finds any errors in your robots.txt file.

Related posts:

1. SEO Site Review – http://www.silvaheeren.com Daniel Silva asked me to comment on the SEOing...
2. Publishing Through A RSS Feed – Quick Guide This is a quick guide to publishing your web...
3. Blocking The Search Engine Spiders From Indexing Specific Pages Dylan Downhill © 2005 Elixir Systems The best way...