Dan Crow, Product Manager at Google, has provided information about “how search engines access and index a website”.


Dan Crow, Product Manager at Google, has provided information about “how search engines access and index a website”.

Crow has answered three questions:

– ‘How does Google know what parts of a website the site owner wants to have show up in search results?

– Can publishers specify that some parts of the site should be private and non-searchable?’.

Crow said: “Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.

In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.” [source]

Post Related Links

Google Blog Post

 

 

 

 

Sharing is caring