The Budget crawl is defined as the limit of pages that Google has set to explore on a website for its referencing. During the exploration, the search engine tries to control its attempts not to overload the server of the site, but while taking care not to ignore the important pages. To find the right balance, Google will therefore assign a different Budget crawl to each website based on some criteria such as: The responsiveness of the server, the size of the site, the quality of content and the frequency of updates
The Web is in full extension, the SEO too. In 2019 alone, more than 4.4 million blog posts were published on the net every day.
While Internet users enjoy consulting this content, search engines are faced with a major challenge: How to organize resources to crawl only relevant pages and ignore spammy or poor quality links
Hence the notion of crawl budget
- What is it concretely?
- How important is it for a website?
- What are the factors that affect it?
- How to optimize it for a website?
- And how to monitor it?
These are some of the questions we will answer clearly and concisely throughout this mini-guide
Let’s go!
Chapter 1: Budget crawl – Definitions and SEO importance
To properly develop this chapter, we will approach the definition of the crawl budget from two angles
- A definition seen from the side of SEO experts
- And what Google really means by this term
1.1 What is the crawl budget?
The crawl budget (or exploration or indexation budget) is not an official term given by Google
Rather, it originates from the SEO industry where it is used by SEO experts as a generic term to designate the resources allocated by Google to crawl a site and index its pages
In simple terms, the crawl budget of a website refers to the number of pages that Google expects to crawl on the site within a given time frame
It is a predetermined number of URLs that the search engine orders to its indexing robotgoogleBot, to consider when it visits the site
As soon as the limit of the crawl budget is reached, GoogleBot stops exploring the site, the rest of the non-indexed pages are automatically ignored and the robot leaves the site
This number of pages to index can vary from one site to another because not all sites have the same performance or the same authority. A fixed crawl budget for all websites could be too high for one site and too low for another
The search engine has therefore decided to use a number of criteria before setting the time that its indexing robot on each site and the number of pages to crawl
Regarding these criteria, there is not an exhaustive list, as for many things related to thealgorithm algorithm
Nevertheless, here are some of these criteria known by all
- The performance of the site a well optimized website with a fast loading speed will probably get a higher crawl budget than a site that is slow to load
- The size of the site the more content a site contains, the higher its budget will be
- The frequency of updates a site that publishes enough quality content and regularly updates its pages will get a considerable crawl budget.
- The links on the site the more links a site contains, the higher its crawl budget will be and the better its pages will be indexed.
Ideally, the crawl budget should cover all pages of a site, but unfortunately, this is not always the case. That’s why it is important to optimize the above mentioned criteria to maximize your chances of getting a large crawl budget
It must be said that until now, the definition of this term has been approached from the point of view of a site owner or a web SEO expert who is in charge of working on the SEO of a website
But, when the question of “what is budget crawl” was addressed to Gary Illyes, an analyst at Google, his answer was much more nuanced
1.2 What does Google really mean by budget crawl?
In 2017, Gary Illyes addressed the question by detailing how Google understands budget crawl In his answerin his answer, we could distinguish 3 main important parts
- The concept of crawl rate
- The demand of exploration
- And some other factors
1.2.1. The crawl rate
During its exploration, Google fears to overwhelm each site crawled and blow up its server. This is not surprising, as we know that the search engine has always been protective and advocates a good user experience on websites
The words of Gary Illyes confirms this once again
“Googlebot is designed to be a good citizen of the Web. Crawling is its top priority while making sure it doesn’t degrade the experience of users visiting the site. We call this the ‘crawl rate limit’, which limits the maximum retrieval rate for a given site.”
So if GoogleBot sees signs on a site that its frequent crawling could affect the site’s performance, then the indexing robot slows down its crawl time as well as its visit frequency
As a result, in some cases, some pages of a website can end up not being indexed by GoogleBot
Conversely, if the indexing robot receives encouraging signs such as fast responses from the server, it facilitates the work and GoogleBot may decide to increase the duration and frequency of its visits
1.2.2. The request for exploration
Regarding the demand for exploration, Gary Illyes explains
“Even if the limit of exploration speed is not reached, if there is no demand for indexing, there will be low activity of Googlebot”
Source: Tech Journey
Simply put, GoogleBot relies on the crawl demand of a site to decide if the site is worth revisiting
This demand is influenced by two factors
- URL popularity the more popular a page is, the more likely it is to be crawled by the crawlers.
- Obsolete URLs google also tries not to have obsolete URLs in its index. So the more a site contains old URLs, the less its pages will be indexed by GoogleBot
Google’s intention is to index recent or regularly updated contents of popular sites and to leave out obsolete ones. This is its way of keeping its index always up to date with recent content for its users
The other factors include everything that could improve the quality of the contents of a site as well as its structure. This is an important aspect that Gary Illyes wanted to focus on
Moreover, he recommends that webmasters avoid certain practices such as
- The creation of poor quality content;
- The implementation of certain style of navigation to facets;
- The duplicated content ;
- And so on
We will come back to these factors in more detail in a later chapter. According to Gary Illyes, by using such practices on your site, you are in a way wasting your crawl budget
Here are his words
“Wasting server resources on pages like these will drain crawl activity from pages that are actually valuable, which can significantly delay the discovery of quality content on a site.”
To better understand what Gary is getting at, let’s say for example that you have a popular e-commerce store that specializes in selling video game consoles and all the accessories
You decide to create a space on the site where your users can share their experiences with your products, a sort of large and active forum. What do you think will happen?
Well, because of this forum, millions of low-value URLs will be added to your link profile. Consequence: These Urls will unnecessarily consume a good part of the crawl budget that Google has allocated to you. As a result, pages that really represent value for you may not be indexed.
Now, you have a pretty clear idea of what the crawl budget is, but concretely how important is it for a website?
Chapter 2: The importance of the crawl budget for SEO
An optimal crawl budget represents a great importance for websites since it allows to quickly index important pages
As Gary Illyes pointed out, a waste of your crawl budget will prevent Google from analyzing your site efficiently. The indexing robot could spend more time on unimportant pages at the expense of the ones you really want to rank for
This would reduce your ranking potential since each non indexed will not be present in Google search results. This page will then lose all probability of being consulted by Internet users from a SERP
Imagine producing interesting content with beautiful and illustrative visuals. Then you go even further by optimizing the Title, Alt, Hn tags and many other factors, just like a knowledgeable SEO expert would do
But if in the end, you ignore the optimization of the crawl budget, I am afraid that Google’s indexing bots won’t reach this quality content and neither will the users. Which would be a shame considering all the SEO efforts that have been made
However, it should be noted that optimizing the crawl budget is particularly important only for large websites
John Mueller, another Google analyst mentioned in a tweet in these words
“Most sites never need to worry about this. It’s an interesting topic, and if you’re crawling the web or running a multi-billion URL site, it’s important, but less so for the average site owner.”
As you can see, the crawl budget is for large websites with billions of URLs like the famous online shopping platforms for example
These sites can really see a drop in traffic if their crawl budget is not well optimized. As for other sites, Google remains less alarmist and reassures that they should not worry too much about problems related to the crawl rate
If you are a small local site or you have just created your website, this is news that makes you breathe a sigh of relief
However, you don’t lose anything by wanting to optimize your crawl budget, especially if you plan to expand your site in the future
On the contrary, many of the methods used to optimize the crawl rate are also used to optimize the ranking of a site on Google pages
Basically, when a company tries to improve their site’s crawl budget, they are also improving their position on Google
Here’s a list of some SEO benefits that a web site gets by optimizing its crawl budget, even if it doesn’t have any problems in this area
- Getting GoogleBot to crawl only the important pages of the site
- Have a well structured sitemap
- Reduce errors and long redirect chains
- Improve site performance
- Updating the content of the site
- And so on
With all these advantages, there is no doubt that the crawl budget is of great importance for a website
But in order to succeed in its optimization, I want to present you some negative factors that you should pay attention to before starting
Chapter 3: Factors that affect the crawl budget
Here are some factors that can negatively affect the crawl budget of a site
3.1. Faceted navigation
This is a form of navigation that is often found on online store sites. The faceted navigation allows the user to choose a product and then have other filters on the page to choose the same product
It can be a product available in several different colors or sizes. The thing is, this way of organizing the results requires adding URLs to the page
And each new URL added just displays a snippet of its original page, not new content. This is a waste of the crawl budget for the site
3.2. URLs with session IDs
The technique of creating session identifiers in URLs should not be used anymore, because with each new connection the system automatically generates a new URL. This creates a massive amount of URLs on a single page
Source Pole Position Marketing
3.3. Duplicate contents
If there is a lot of identical content on a site, Googlebot can crawl all the pages involved without coming across any new content
The resources used by the search engine to analyze all these identical pages could be used to crawl URLs with relevant and different content.
3.4. Software error pages
Unlike 404 errors, these errors do point to a functional page, but not the one you want. Since the server cannot access the actual content of the page, it returns a 200 status code
3.5. Pirated pages
Google always makes sure to provide the best results to its users, which of course excludes hacked pages. So if the search engine spots such pages on a site, the indexing is stopped to prevent these pages from ending up in the search results.
3.6. Infinite spaces
There are infinite spaces that can be found on a site. Let’s take the case of a calendar where the “next month” button is programmed to actually send to the next month
When crawling, Googlebot could fall into a kind of infinite loop and empty the site’s crawl budget by constantly searching for “next month”
Every time a user clicks on the right arrow, a new URL is created on the site, making for an astronomical number of links on the one page
3.7. Spammy or low quality content
Google tries to have in its index only quality contents that bring a real added value to Internet users
So, if the search engine detects poor quality content on a site, it gives it very little value
3.8. Redirect chains
When for one reason or another, you create chains of redirect chainsgooglebot has to go through all the intermediate URLs before reaching the real content.
The robot still can’t tell the difference, so it treats each redirect as a separate page, which unnecessarily drains the crawl budget allocated to the site.
3.9. The cache buster
Sometimes files like the CSS file can be renamed like this: style.css?8577484 The catch with this naming scheme is that the “?” sign is automatically generated every time the file is requested.
The idea is to prevent users’ browsers from caching the file. This technique also generates a massive amount of URLs that Google will process individually. This affects the crawl budget.
3.10. Missing redirects
In order not to lose a part of its audience, a site can decide to redirect its non-www version to a www version exactly like twaino.com to www.twaino.com or vice versa
It may also want to migrate from hTTP protocol to HTTPS. However, if these two types of general redirects are not properly configured, notice that Google might crawl 2 or 4 URLs for the same site:
http://www.monsite.com
https://www.monsite.com
http://monsite.com
https://monsite.com
3.11. Bad paging
When the pagination is just composed of buttons that allow navigation in both directions, without any numbering, Google needs to explore several pages in this direction before understanding that it is a pagination
The pagination style that is recommended and does not waste the crawl budget is the one that explicitly mentions the number of each page.
With this pagination style, Googlebot can easily navigate from the first page to the last page to get an idea of which pages are important to index.
If you detect one or more of the factors mentioned so far, it is important to optimize your crawl budget to help Googlebot index the most important pages for you.
But how to optimize it?
Chapter 4: How to optimize the crawl budget?
Now that the basics have been established, we can start optimizing the crawl budget itself
4.1. Allow crawling of your important pages in the Robots.Txt file
This first step of the budget crawl optimization is probably the most important of all
You have two ways to use your robots.txtfile, either directly by hand or with the help of an SEO audit tool
If you’re not a programming buff, the tool option is probably the easiest and most effective
Take a tool of your choice, then add the robots.txt file. In just a few seconds, you can decide to allow or disallow crawling of any page on your site
Find out how to do it in this guide entirely devoted to robots.txt
4.2. Avoid redirect chains as much as possible
You can’t definitely avoid using redirect chains because in some circumstances, it is an approach that can be useful. However, ideally, you should only have one redirect on a website
But, admittedly, this is almost impossible for large sites. At some point, they are forced to use 301 and 302 redirects
However, setting up redirects on the site at all times will severely damage the crawl budget and even prevent Googlebot from crawling the pages you want to be listed for.
Be content with just a few redirects, one or two and preferably short chains
4.3. Use HTML code as much as possible
Google’s crawler has gotten much better at understanding content in Javascriptbut also Flash and XML
But other search engines can’t say the same. So to allow all search engines to fully explore your pages, it would be wiser to use HTML which is a basic language, accessible to all search engines
Thus, you increase the chances of your pages being crawled by all search engines
4.4. correct HTTP errors at the risk of losing your crawl budget unnecessarily
If you didn’t know it, the 404 errors and 410 errors literally eat away at your crawl budget. Even worse, they degrade your user experience
Imagine the disappointment of a user who rushes to check out a product you sell or content you’ve written and ends up on a blank page, with no other alternative
This is why it is important to correct HTTP errors for the health of your site, but also for the satisfaction of your visitors
As for redirection errors, it would be more practical to use a dedicated tool to spot these errors and fix them if you are not very fond of programming
You can use tools like SE Ranking, Ahrefs, Screaming Frog.. to perform an SEO audit of your site
4.5. Set up your URLs properly
If you set up different URLs to access the same page, Google will not understand it that way. For the search engine, each page is identifiable by one and only one URL
And so Googlebot will index each URL separately, no matter if there are several of them redirecting to the same content. This is a waste of the crawl budget
If you have such a situation on your site, you should notify Google
This is a task that has a double advantage, on the one hand your crawling budget is saved and on the other hand you no longer have to fear a penalty from Google for your duplicate content
Find out how to manage your duplicate content by consulting this article which shows in detail the procedure to follow
4.6. Keep your sitemap up to date
It is important to keep your XML sitemap up to date. This allows crawlers to understand the structure of your site, making it easier to crawl
It is recommended to use only Canonical URLs for your sitemap
4.7. Hreflang tags are essential
When it comes to crawling a local page, crawlers rely on hreflang tags. This is why it is important toinform Google of the different versions you have
Find all the information about the Hreflang tag in this mini-guide.
That was 7 tips to optimize your site’s crawl budget, but it doesn’t end there. It is also useful to monitor how Googlebot crawls your site and accesses your content
Chapter 5: How to monitor your crawl budget?
To monitor the crawl budget of your site, you have two tools to do so Google Search Console and the server log file
5.1. Google Search Console
GSC is a tool proposed by Google that offers several features that allow you to analyze your position in the index and SERPs of the search engine. The tool also offers an overview of your crawl budget
To start the audit, you can access the report ” Crawl Statistics “. This is a report that traces the activity of googlebot on your site over the last 30 days
On this report, we can see that Googlebot crawls daily an average of 48 pages on the site. The formula to calculate the average crawl budget for this site would be
BC = Average number of pages crawled per day x 30 days
BC = 48 x 30
BC = 1440
So for this site, the average crawl budget is 1440 pages per month
However, it’s important to note that these Google Search Console numbers are a bit of a stretch
But in the end, this is an estimate and not an absolute value of the crawl budget. These numbers just give you an idea of the crawl rate the site is receiving
As a little tip, you can write down this estimate before you start optimizing your crawl budget. Once the process is complete, you can look at the value of this crawl budget again
By comparing the two values, you will better evaluate the success of your optimization strategies
Google Search also offers another report, a “coverage report” to know how many pages have been indexed on the site, as well as those that have been de-indexed.
You can compare the number of indexed pages to the total number of pages on your site to know the real number of pages ignored by Googlebot.
5.2. The server log file
The server log file is undoubtedly one of the most reliable sources to check the crawl budget of a website
Simply because this file faithfully traces the history of the activities of the indexing robots that crawl a site. In this log file, you also have access to the pages that are regularly crawled and their exact size
Here is an overview of the contents of a log file
11.222.333.44 – – [11 / March / 2020: 11: 01: 28 -0600] “GET https://www.seoclarity.net/blog/keyword-research HTTP / 1.1” 200 182 “-” “Mozilla / 5.0 Chrome / 60.0.3112.113 “
To interpret this piece of code, we’ll say that on March 11, 2020, a user tried to access the page https://www.seoclarity.net/blog/keyword-research from the Google Chrome browser
The code “200” is set to mean that the server did access the file and that the file is 182 bytes in size
Chapter 6: Other questions asked about the Budget Crawl
6.1. What is the crawl budget?
The crawl budget is the number of URLs on a website that a search engine can crawl in a given period of time based on the crawl rate and the crawl demand.
6.2. What are crawl rate and crawl speed limit?
The crawl rate is defined as the number of URLs per second that a search engine can crawl on a site. The speed limit can be defined as the maximum number of URLs that can be crawled without disrupting the experience of a site’s visitors.
6.3. What is crawl demand?
In addition to the crawl rate and crawl speed limits specified by the webmaster, the crawl speed varies from page to page depending on the demand for a specific page.
The most popular pages are likely to be crawled more often than infrequently visited pages, pages that are not updated, or pages that have little value
6.4. Why is the crawl budget limited?
The crawl budget is limited to ensure that a website’s server is not overloaded with too many simultaneous connections or a high demand on server resources, which could negatively impact the experience of website visitors.
6.5. How often does Google crawl a site?
The popularity, crawlability, and structure of a website are all factors that determine how long it will take Google to crawl a site. In general, Googlebot can take more than two weeks to crawl a site. However, this is a projection and some users have claimed to be indexed in less than a day.
6.6. How to set a crawl budget?
- Allow crawling of your important pages for robots;
- Beware of redirect strings;
- Use HTML whenever possible;
- Don’t let HTTP errors consume your crawl budget;
- Take care of your URL parameters;
- Update your sitemap;
- Hreflang tags are vital;
- Etc.
6.7. What is the difference between crawling and indexing?
Crawling and indexing are two different things and this is often misunderstood in the SEO industry. Crawling means that Googlebot examines all the content or code on the page and analyzes it
Indexing on the other hand means that Google has made a copy of the content in its index and the page can appear in Google SERPs.
6.8. Does Noindex save the crawling budget?
The Noindex meta tag does not save your Google crawl budget. This is a pretty obvious SEO question for most of you, but Google’s John Mueller confirmed that using the noindex meta tag on your site will not help you save your crawl budget. This meta tag only prevents the page from being indexed, not crawled.
Conclusion
In short, the crawl budget is the resources that Google reserves for each site for the exploration and indexation of its pages
Even if the crawl budget is not in itself a factor of referencing, its optimization can contribute to the referencing of the pages of the site, especially the most important pages
For this, we have presented throughout this guide a number of practices to take into account
I hope this article has helped you to better understand the concept of budget crawl and especially how to optimize it
If you have any other questions, please let me know in the comments.
Thanks and see you soon!
