Google & WordPress Robots.txt Handling Is Being Looked Into

One of the takeaways from the Google Webmaster Conference was that if Google tries to access your robots.txt file is unreachable but it does exist then Google won't crawl your site. Google said about 26% of the time GoogleBot cannot reach a robots.txt file. WordPress might make changes in order to reduce this error rate.