Crawl Budget

Want a website to appear as high as possible in Google or other search engines? At least you’ll need enough crawl budget. What is crawl budget and in what ways do you get the most out of it for your website? I’m happy to explain that to you here.

What is crawl budget?

Crawl budget represents the amount of pages a search engine indexes on your website per day for content. Bots scan your website and this is how a search engine determines how relevant your website is. Every website has its own crawl budget.

How a crawl budget works

The crawl budget of a website is different for everyone. This depends in part on how big the website is, how often other sites link to your website and how fast your website is. With Google Search Console, it is possible to find out your website’s crawl budget.

It is better when a search engine crawls your website regularly. This ensures that your website remains relevant in search results, boosting online findability. Therefore, it is smart to optimize your crawl budget. This is possible by making your website faster and by providing more links.

The impact of crawl budget on SEO

A crawl budget certainly has an impact on the SEO of a website or page. The higher the crawl budget, the better: your website then experiences regular checks by bots. This keeps your website relevant to the search engine and has a positive effect on organic findability. Therefore, it is smart to make the right changes so that you get the best results.

How do I determine how much crawl budget a website has?

Often, as an SEO specialist, I have a quick gut feeling of how much crawl budget a website actually has. I like to express this in how long does it take for Google to pick up on a change to the website (depending again on where that change occurs in the website). A change on the front end of the website (think home page) will always be picked up faster than a change on the back end of a website.

The easiest way to measure the speed of a change is to change a page title on a page of your choice and simply measure when the change actually shows up in the SERP.

Measuring crawl budget through the Google Search Console

When you’re in the Google Search Console, you can go to “Settings” on the left menu, then “Crawl Statistics.” At “Crawl Statistics” you’ll find the overview below.

What Google says about all metrics:

Total number of crawl requests: The total number of crawl requests to your site, during the displayed period. Duplicate requests to the same URL will be counted.

Total download size: The total size of all files and resources downloaded during crawling, during the displayed period. If a resource has already been cached during a previous crawl, it will not be counted. The number of bytes displayed includes the file-html, associated image bytes, script files and css files.

Average response time (ms): Average page response time for a crawl request to retrieve the page content. Does not include page source retrieval time (scripts, images and other linked or embedded content) or page rendering time.

Types of crawls

The types of crawls vary from each other and again can be interpreted differently:

  • Per response (status codes)
  • By file type
  • Per goal
  • By Googlebot type

My advice

There are a number of things you can change in order to improve crawl budget. First of all, I recommend speeding up your website: this will make your website more interesting for the search engine and, in addition, it is also better for the user. Speeding up the website is possible in several ways; including using the right images that do not have too much load time.

Also, get more relevant inbound links: link building with trusted parties ensures this and then also offers better results in terms of your crawl budget.

Frequently Asked Questions

What is a Crawl budget?

The crawl budget indicates the number of pages a search engine indexes your website for content per day. Each website has its own crawl budget. The scanning is done by bots and based on the results, the search engine determines the relevance of your site.

What is the impact of Crawl Budget on SEO?

You can argue that the higher the crawl budget the better. You will then be scanned regularly and this way you will remain relevant to the search engine and thus be easily found.

Senior SEO-specialist

Ralf van Veen

Senior SEO-specialist
Five stars
My clients give me a 5.0 on Google out of 75 reviews

I have been working for 10 years as an independent SEO specialist for companies (in the Netherlands and abroad) that want to rank higher in Google in a sustainable manner. During this period I have consulted A-brands, set up large-scale international SEO campaigns and coached global development teams in the field of search engine optimization.

With this broad experience within SEO, I have developed the SEO course and helped hundreds of companies with improved findability in Google in a sustainable and transparent way. For this you can consult my portfolio, references and collaborations.

This article was originally published on 22 March 2024. The last update of this article was on 22 March 2024. The content of this page was written and approved by Ralf van Veen. Learn more about the creation of my articles in my editorial guidelines.