SEO and Crawl Budget Guide

Ignoring the crawl budget can cost you rankings, traffic, and conversions. If your website is not optimized for a maximum crawl budget, no matter the size of the site, crawl activity may be wasted on low value or low performing pages, preventing the pages designed to improve SEO from being crawled.

How can crawl budget impact SEO?

According to Webmaster Central Blog, a crawl budget is “the ‘crawl rate limit,’ which limits the maximum fetching rate for a given site.” 

Translation: Google will not crawl every URL on your website. A crawl budget protects against overloading website servers and helps Google deliver the most popular URLs in the search results. 

In 2017, Google released a statement on crawl budget, “we'd like to emphasize that crawl budget, as described below, is not something most publishers have to worry about. ...if a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently.” 

Given that update, is it safe to ignore crawl budget when the site is set up for small business SEO? NO.

Ignoring the crawl budget can cost you rankings, traffic, and conversions. If your website is not optimized for a maximum crawl budget, no matter the size of the site, crawl activity may be wasted on low value or low performing pages, preventing the pages designed to improve SEO from being crawled. If crawl budget is limited, new or important pages will not be prioritized over older or non-priority pages. Wasting resources on low-value pages (soft error pages, hacked pages, duplicate content pages, low quality/spam pages, URLs with cumbersome session/tracking IDs) will draw crawl activity away from pages that have value, causing delayed indexing on URLs designed to improve SEO (and lead generation). If the pages that are designed for conversions from an Enterprise SEO campaign aren't indexed, the entire strategy will fail to generate traffic that delivers conversions.

The number of pages Google crawls (the budget), is determined by three main factors:

• the size of your site
  
• the technical health of your site
  
• the number of links to your site
  

Follow these steps to effectively manage your crawl budget:

1. Technical SEO Audit

Crawl budget optimization begins with a technical SEO audit. Use a tool like SEMrush, Screaming Frog, or DeepCrawl to check all of these technical items on your site:

• Indexed pages: Make sure all of the pages that are indexed are valuable. If you find pages that have dynamically generated URLs, use a robots.txt file to give crawlers directions on disallowed pages. 
  
• Use rel=”nofollow” to moderate crawl activity to low-value pages.
  
• Identify and fix broken links: Broken links will expand the crawl budget, wasting it on pages that deliver error codes.
  
• Speed: If page load speed is an issue, Google will reduce the crawl rate for your website.
  
• Manage Redirects: While the 301 redirect is the gold standard of SEO, it doesn't make sense when the number of 301 redirects and redirect chains begin to interfere with your crawl budget. In some cases, it’s better to unpublish a blog post and let the 404 code run its course or use a 410 Deleted status code. Yoast has an excellent guide on this process: Update or Delete Old Content On Your Site. If 301 redirects exist outside of the blog URLs, use this guide from Search Engine Journal to help you assess page value and reduce the number of redirects.

2. Reports in Google Search Console

Check the Index Coverage Report in Search Console. Here, you’ll see the status of all the pages on your site, according to Google’s crawler. Pages with errors, warnings, and exclusions should be double checked and corrected. 

3. Sitemap

Double check your sitemap - remove URLs that are not the canonical and make sure it corresponds to the newest uploaded version of robots.txt. 

Final Step: Link Building

Most tutorials and articles about SEO and crawl budget don’t mention link building, but they should. Crawl demand is an important SEO ranking factor in a crawl budget. 

Learn more about building demand for your website through link building from SEMrush: What Are Backlinks?

What about website hosting?

Even if your website uses a host that has fast processors and has the resources to sustain a high crawl rate (like SiteGround), it doesn’t guarantee that Google and other search engines will prioritize those pages over your competitors. If you’re investing in SEO as part of your long-term digital marketing strategy, don’t ignore crawl budget optimization. Even if your site is less than a few thousand URLs, optimizing the crawl budget will ensure that your SEO efforts pay off in the long run, delivering ROI for your content and link building campaigns.

Conclusion

The web has grown exponentially since the early 1990s:

Source: Statista

At the end of 2019, there were 1.72 billion websites online. If you want to rise above your competitors from your SEO efforts, spend your crawl budget wisely.