Site submit to Google

Is Googlebot the same as "Add an URL" or how does it Work?

Just as in the beginning when Sergey Brin and Larry Page first started Google, the search engine runs on several low-cost computers – only this time thousands – for faster processing. Analogous processing is a technique of calculations in which many different computations can be preformed concurrently which will speed up the data processing considerably.

There are three parts to the Google network that works together to make it function properly. The fist is the Googlebot, a web sycophant that will find and retrieve web pages. Next is the Indexer that sorts every page and every word for the database and finally there is the query processor that will compare a search to the index and make a recommendation it feels is most relevant.

The Googlebot is specifically designed for Google and is its web crawling robot that will go out and find pages on the Internet and hand them to the Indexer. Many people think of the Googlebot as an actual robot spider scurrying across a web of information, plucking out what it needs and running back to the ‘base’ to unload. Much like a web browser, Googlebot sends a request to a web server for a web page downloads the page and then gives it to the Indexer.  

Googlebot is made up of many different computers all making requests and retrieving webpages, sometimes simultaneously, much quicker than an individual could with their web browser. Googlebot deliberately makes its requests slower to avoid overwhelming servers with slower capacities.  

There are two ways in which a Googlebot will find URLs and pages: through feeling around on the Internet to find links and through an ‘add an URL form.’

For every good there is a bad and the bad in this case would be spam. The spammers came up with a way to flood the ‘add an URL form’ with thousands and thousands of URLs directing searchers to commercial advertising.  Google will reject those URLs it believes are trying to deceive website visitors by using such dirty tactics as stuffing a page with irrelevant words, including hidden links and text on a page, bait and switching or cloaking, using sub-domains with significantly comparable content or using sneaky misdirects. The ‘add an URL form’ now contains a task for a live person to complete – type in these letters or numbers in the space provided and it will automatically ‘kick out’ any of the computer generated ‘guessers.’ 

Deep crawling is when the Googlebot retrieves a webpage and collects all the links from it and places them in a ‘holding’ area for later. Many think that the Googlebot runs into a lot of spam but that is not the case because most people build their webpages with high quality content and links to other webpages with good content. Because the Googlebot harvests every link it encounters, it will come up with a large database of a variety of items on the web.   

The Googlebot ‘recrawls’ webpages that are popular and updated regularly in order to keep the popular search engine up-to-date and the most relevant information available to it’s searchers. Newspaper pages are downloaded on a daily basis however pages with stock quotes are downloaded even more frequently then that, these are known as ‘fresh crawls.’ Deep crawls reveal more pages than fresh crawls. The combination of the two crawls is what keeps Google reasonably current.