Chris Boggs’ latest blog post is titled “How Does Googlebot Find/Index Hidden FTP Logs?”.


Chris Boggs’ latest blog post is titled “How Does Googlebot Find/Index Hidden FTP Logs?”.

Boggs says, “Google’s crawler constantly scours the Internet for pages to index, which is one of the reasons you should run away if someone offers to “submit your website to Google.” On any page you do not want indexed, it is important to disallow the Googlebot (one of the nicknames for their spiders) by using special code”. [source]

Go to SEO Round Table to read the complete post.