Google Reveals Some GoogleBot Secrets
Google has revealed some secrets about how and why it crawls our websites.
The insight was revealed at SMX East, a massive search marketing conference taking place in New York this week.
An analyst working with Google Webmaster Trends, Gary Illyes, disclosed that GoogleBot will stop visiting a website and crawling it in some situations. These include:
- Being unable to access a website. If GoogleBot can’t connect to your server, it will strike you off the list
- Detecting a very slow response from a website. Again, GoogleBot will reduce the frequency of visits if it finds that connecting takes a long time. The reasoning is that GoogleBot could potentially take a site offline if it’s responsible for slowing it down
- Getting a 5XX error. 5XX errors generally mean a site is struggling to respond, or experiencing some kind of catastrophic fault that is preventing it from loading
Interestingly, a few days ago, John Mueller said that GoogleBot doesn’t care too much about broken links. At least we know one factor won’t stop it crawling.
GoogleBot visits are good. Everyone who works on their Search Engine Optimization (SEO) wants to see GoogleBot visiting frequently so that fresh, new content is always indexed as quickly as possible.
If GoogleBot detects errors and slows down the frequency of its visits, content is not going to be indexed, and your content marketing efforts will be wasted.
The bot isn’t everyone’s friend. Some webmasters have found that it can overwhelm their site with requests. But if your server can cope with the requests, it should be welcoming this particular spider with open arms.
To ensure your site is fast enough for GoogleBot, make sure to optimize your website properly and use a high performance web host. Use these tips:
- Choose a host with a data center that is geographically close to you and your visitors
- Ping your current host to check response times before you sign up for a new hosting account
- Upgrade from shared hosting to a Virtual Private Server (VPS) so you are less likely to be dragged down by other sites
For more information on speed, check out our reviews on the best performing web hosts.