The Shark Tank

Google has issued a warning to website owners and content delivery networks against using 4xx client server status errors, such as 404s and 403s, to limit Googlebot crawling.

What is Googlebot?

Googlebot is a program used by Google to scan and index web pages. It works by following links from one page to another, cataloging each page’s content, and storing it in Google’s database to include it in search results. In other words, Googlebot is responsible for crawling, indexing, and ranking web pages in Google’s search results.

Why the Notice

Google recently observed a rise in the number of websites attempting to use 4xx client server status errors, such as 403s, to restrict Googlebot crawling. This technique involves website owners intentionally signaling to Google’s crawler that specific pages or content on their site should not be crawled or indexed. In response to this trend, Google warned against using these error codes for crawl rate limiting purposes.

What to Do Instead

Instead of using 404 and other 4xx client server status errors to limit Googlebot crawling, Google recommends using the Google Search Console crawl rate settings to adjust your crawl rate. The Google Search Console crawl rate settings allow website owners to adjust the crawl rate of Googlebot, which determines how often it crawls a website and how much bandwidth it uses.

Using these settings, website owners can optimize Googlebot’s crawl efficiency and ensure that their website is crawled as intended. Making changes to these settings is a quick way to reduce the crawl rate, and the changes are generally reflected within days.

How to Reduce Crawl Rate

Google warns website owners to avoid setting the crawl rate too low for their site’s needs and to learn more about what crawl budget means for Googlebot. If changing the crawl rate is not possible, Google recommends reducing the crawl rate for a short period of time, such as a few hours or 1-2 days, and returning an informational error page with a 500, 503, or 429 HTTP response status code.

If you’ve noticed that Googlebot is not crawling your site as frequently as it should, it could be due to your hosting provider or content delivery network (CDN) using these error codes to reduce crawl rates. You can always submit a request to Google support to ensure that you are not using 404s or 403s to reduce crawl rates. This is important for maintaining your website’s visibility and search engine ranking.

Call an SEO Expert for Website Management Help

As a business owner looking to grow your online presence, it is important to be aware of Google’s guidelines and recommendations for website management. By following best practices, you can ensure that your website is optimized for search engines, making it easier for potential customers to find you online. If you need assistance with website management or SEO, consider working with a digital marketing agency like BluShark Digital to help you achieve your online goals.