Indexing in SEO refers to the procedure by which a search engine lists, stores and orders the pages of a website to quickly display them when a user launches a search. It is an essential step in natural referencing without which a website cannot be displayed in the results of a search engine
Search engines like Google or Bing are able to provide thousands of answers to a query in only a fraction of a second. But behind this speed of processing, there are many steps includingindexing
So
- What is indexing?
- How important is it for SEO ?
- And how to facilitate the indexing of a site?
Discover the answer to all these questions and more in this mini-guide entirely dedicated to indexing
Let’s get started!
Chapter 1: Indexing – How it works and why it’s important for SEO
Before we overwhelm you with technical terms, it’s appropriate that we cover how this concept works to avoid any ambiguity in the rest of the guide
1.1) How does indexing work?
When a website is newly created or a page is newly added to a site, they are not automatically accessible to search engines. The engines must first go through a number of steps before they can find them
And to reach these pages, search engines will send what is called in SEOcalled crawlers. As its name indicates, a crawler explores sites by tracing a path based on internal links and external links
Through these links, the crawler will “travel” between the pages of a site or from one site to another
During this journey of exploration, the robot will collect information, order it and store it in a database: this is the indexing stage. This database is called ” Index ”
After a request is launched, the search engines will draw on the information stored in the index to propose answers based on the keywords entered by the user in his request.
Without this indexing step, no page of a website can be displayed in the search engine results after a search. It is an essential step in natural referencing, but not sufficient, to rank a page well on search engines
That is to say that to succeed in having a good ranking on Google or any other search engine, you must gather other ranking criteria after the indexing stage of the pages
So, when a user launches a query on Google or Bing for example, he does not launch his search directly on the web, but rather in the index of the engines
If the search is related to your theme, your page will have no chance to be presented to the user if it has not been previously explored and indexed
Moreover, it must be said that search engines do not always index all the pages available on the Web. Sometimes, because of poor quality content or bad SEO practices, some pages of a site may not be indexed
We will come back in more detail in a later chapter on the factors that can prevent the indexing of a page
But once the user launches his request, what actually happens between the time that separates his request and the response of the search engine?
Well, the search engine will
- Analyze the user’s request to understand his search intention;
- Search its index and filter the information according to the user’s intention;
- Select all the pages deemed relevant to the request on the basis of several ranking criteria ;
- Then display these pages to the user in order of relevance
While some sites regularly update their pages, others end up obsolete
Also new pages are created every day, bringing if possible new information more relevant than the old contents
Faced with this continuous change of information, to remain efficient indexing robots are forced to regularly revisit the sites already indexed
The frequency of visit of a crawler on a site depends on several factors that define the Crawl budget of the site
1.2. how does indexing work with Google?
As previously explained, Google also works in the same way to index its web pages. What we must specify is that to explore the web and index the available pages, Google uses its indexing robot called Googlebot
If during the exploration, Googlebot comes across a page that is optimized for indexing, the robot can try to understand the theme it addresses
It can be a newly created page or an old page, as long as it meets the criteria for indexing optimization
To begin this indexing phase, Googlebot will analyze the page’s content, catalog the inserted visuals and any other available data that can help it understand the page’s purpose
All the information collected will then be classified by theme and stored in the Google index
This Google index is a vast database that contains several hundred billion pages. The search engine itself estimates the size of its index at over 100 million gigabytes
Now, let’s take a look at the importance of indexing for your site’s SEO.
1.3. How important is indexing to SEO?
When we explained how search engines index web pages, we mentioned the first importance of indexing for the SEO of your site
You already know that without the storage phase in the web index, the pages of your site can not be visible by Internet users on a results page of engines
And that’s not all, when Googlebot visits your site, the robot may detect malware, technical problems or poor quality content
In other words, you can take the Google index as a reliable source. If some of your pages are listed there, it is a proof that these pages are “healthy” and that they are relevant enough to be presented to Internet users
However, remember that Google also has a secondary index where it stores pages of lesser quality. These include duplicated pages or deemed “less relevant” by the search engine
In principle, Google does not display the pages ranked in the secondary index in its results, because it does not give the same importance to all pages. Ideally, your web pages should be ranked in the main index
But why are some pages not indexed by search engines? We’ll talk about this in the next chapter
Chapter 2: Factors that prevent the indexation of a page
Before presenting the factors that could block the indexing of your pages, here is what you incur if it really happens
2.1. The consequences of indexing problems
The nature and the problems that affect the indexing of a page are not the same in terms of consequences on the site
If it is a page that is less important to the owner that is affected, it is certainly not the end of the world
But if on the other hand, it is an important page with a highly optimized content that is affected, it is a pity and all the SEO efforts go down the drain.
If it is an e-commerce store, the consequences will be even heavier. It is agreed that for these e-commerce sites, organic traffic is more profitable in the long run than advertisements or PPC campaigns
Imagine that a good part of the product listings of the company are not indexed by Google. Consumers will not see them and the store will see its conversion rate considerably
Now, let’s look at the actual factors:
2.2. 10 common reasons why Google does not index all links
It is true that we have defined the index as the database of the engines, but the indexing robots are very selective and do not store everything in their path
There are common reasons why search engines do not index all links
2.2.1. Pages that return response codes other than 200
You may not know this, but if the pages on your site do not return the 200 response code, there is no chance that they will be indexed or continue to be indexed if they have been indexed
The code 200 (OK) indicates a positive response from the server following a request to access a page
For reasons of redirections redirections, access to a page may return 404 errors and until the error is cleared, search engines will not index this page
You can check the status of your important pages on HTTPStatus.io. Just the URL of the page and the tool displays the status of the page
If you find pages with the 404 code, I invite you to consult my article 404 error: Why and how to correct it efficiently?
2.2.2. Indexing can also be blocked by the Robots.txt file
You may invest the same attention to write all your content, but there are probably pages that represent more interest for your business than others
Even if the challenge for search engines is to provide relevant pages to Internet users, what would be your interest if they prioritize pages that are not necessarily a priority for you?
That’s why search engines give website owners the power to indicate which content they actively want to be indexed
The robots.txt file file is located at the root of your site and allows you to make indexing recommendations to search engines. So if one of your pages is not in the Google index, the first thing to do is to consult the Robot.txt file
If it is a page already indexed by the search engine, you will receive a message notifying you that the page is no longer available and that the problem comes from the Robots.txt file:
Source Kern Media
See my article on the Robots.txt file to find out how to optimize your Robots.txt file and fix such errors.
2.2.3. The Meta Robots tag with the value “Noindex
Another one of the most common reasons why a website can end up unindexed by Google is the presence of the Meta Robots tag in the part of the page’s source code
If this tag takes the value “noindex”, it is a message to Google telling it not to index the page concerned. And indeed, Google will not index the page until the value is changed
To check the value of the meta Robots tag tag on a page, right-click on the page and click on “Inspect Element” to access the source code directly
To go fast, you can combine the keys ” Ctrl + Uto access the source code
Once in the code, replace the value of the “content” parameter with the one that suits you best. Discover the syntax of the tag as well as the different possible values to assign in this article which defines the meta Robots tag.
2.2.4. The X-Robots tag with the value “Noindex
This tag works a bit like the Meta Robots. It also allows you to control the way Google should index a page. But it must be said that the X-Robots tag is rather found in the response of the header of web pages or certain documents
These are usually non-HTML pages without the section such as PDF, DOC, etc
Unless you intentionally added “noindex” to the X-Robots tag, it is very rare for this to happen by accident. In any case, check this eventuality as well to make sure that it is not the X-Robots tag that is preventing your page from being indexed
To do this, you can use the chrome extension SEO Site Tools :
Source Kern Media
2.2.5. Duplicate content on the site
The duplicate content on the same site are very harmful for SEO in general. A content that is partially or completely duplicated on another page of the same site can prevent the others from being indexed by search engines
So if you notice duplicate content on your site, it may be the root of your indexing problem. If the number of copies is high, even the original page will be downgraded on Google SERPs
To find out if you have duplicate content on your site, you can use the Siteliner to quickly crawl the site and retrieve the URLs of duplicate pages
It’s quite simple to use, you just enter the domain name of the site and the tool will provide a graph with the percentage rate of duplicated content.
Google remains quite firm on duplicate content and tolerates a few small passages that may be repeated
So to a certain extent, it is quite possible that pages with similar content on your site can be indexed and get a ranking on Google
But if it is a question of a large volume of content that is copied and pasted entirely onto other pages, it is likely that Google will penalize these pages and even remove them from its index
2.2.6. Duplicate contents outside the site
Even if you are careful with duplicate content on your site, duplicate content coming from outside, i.e. from other sites, can still reach you
A large amount of duplicate content on several other sites would also look bad to the search engine
Regardless of your industry or the type of site you have, Google’s penalties for duplicate content remain the same. If you suspect that some websites have plagiarized an excerpt of your content, put the excerpt between 2 quotation marks and then run a search on Google
You will get a list of websites that have used this excerpt in their content, similar to the way citations are found on Google
To be sure of the result, you can use a dedicated tool capable of spotting duplicate content, it is Copyscape. The tool provides a detailed report of the websites that have plagiarized you
You can also use the plagiarism checker Quetext plagiarism checker to find out if sites have scrupulously copied your content
Once these sites are listed, find out in this article strategies to claim your duplicated contents and ask Google to give you back the right
2.2.7. Pages that do not add value to Internet users
The first challenge of search engines is to provide results that are relevant and that respond effectively to the user’s query
Unfortunately, not all content on the Net provides real added value to Internet users.
If you think you are in the same situation, fix the quality of your content as soon as possible to offer a better experience to your users. Google will reward you by indexing your pages regularly
We have for example theaffiliation sites that usually generate ads without caring too much about the satisfaction of the user. Google’s algorithms are becoming more and more intelligent and are able to detect these pages of little value and not index them.
2.2.8. Your site has just been created
If you have just created your site, it is important to know that its indexing will not be systematic. It takes time for Google or any other search engine to discover you.
As explained above, to reach your site, Google will make its way with links. This is why it is important to have good link building strategiesespecially if it is a newly created site
2.2.9. The speed of loading your page
Beyond the quality of the textual content, a site must also improve its technical performance
In spite of its contents, a site that takes forever to load, or that chains pop-up windows in an untimely manner or that offers a bad user experience will always be badly seen by search engines
The loading time of your pages directly impacts the traffic rate. The slower the loading time, the less time users will spend on your site
And since Google always advocates user experience, the engine discourages sites that do not have a fast loading speed
If the slowness problem persists, Google may even decide to remove the affected page from its index
There are a number of tools to test the loading speed of your pages, including Page Speed Insights from Google or the tool GTMetrix
2.2.10. Orphan pages
To update its index, Google regularly explores websites, including sitemaps XML SITEMAPS. If yours is often visited by Googlebot and you meet the ranking factors, the search engine will be able to improve your positioning on its SERPs
But if during the crawl Google does not find links that redirect to a particular content on your site, that content will simply not be indexed
No matter if the link is internal or from an external site, Google needs it to reach the content, crawl it and if possible index it
These pages that have no internal link are what we call in SEO ” orphan pages “. If your site has too many orphan pages, it can discourage Google from indexing your site regularly
To detect orphan pages on your site, you can use the Screaming Frog
For the little trick, you can export all the URLs crawled by Screaming Frog on a spreadsheet
Then compare this first list of URLs to the one available in your XML sitemap. All links that are in the sitemap and not in the Screaming Frog crawl report will be considered orphan pages
It’s good to know how to recognize the indexing problems that undermine your site and fix them.
But it’s even better to constantly keep an eye on the site to quickly spot these problems before they affect your SEO
2.3. monitor your site’s indexing status with Google Search Console
As we have seen, having indexing errors can ruin a site’s SEO and decrease its revenue
But before that happens, Google provides you with a tool to constantly monitor the indexing status of your site: it is the Google Search Console
To use this tool, first access its home page
Then in the left sidebar, click on the following options located just below the “Index” tab:
- Coverage
- Sitemaps
It is recommended to check the indexing status at least once every month or 2 months to detect errors rather
After fixing the indexing problems of their site, a question often comes up among site owners: “When will my page be indexed?”
2.4. Knowing when a page will be indexed
Many website owners worry and wonder when their page will be indexed. Well, unfortunately there is no precise answer that can be given to this question
The fact is that you don’t control everything when it comes to indexing a page. Despite your best efforts, according to Google’s guidelines, there are other external factors that do not necessarily depend on your willingness to see a page quickly indexed.
Simply because you are not the only one who wants to be indexed by Google. There are millions and millions of other pages available on the Web waiting to be explored
Your turn may be tomorrow, or next week, or in a few months, it is difficult to give an exact difficult to give an exact time frame. Everything will depend on the frequency that Google has set to explore your site.
Nevertheless, Google offers the possibility to request the crawling of your new URLs
2.5. How to request a crawl to Google ?
For any new or recently updated page on your site, you can formulate an indexing request and send it to Google by following the methods below
2.5.1. Use the URL inspection tool to request indexing of some pages
Before presenting the procedure to follow, it is important to note that without access to a Google Search Console account, you cannot request an indexation with the Google inspection tool
To do this
- Monitor the relevant URL with theuRL inspection tool inspection tool
- Select “URL Inspection”. The tool will start an online test of the URL to check if it is already indexed or not:
You will be notified if there is any problem or if the page is already present in the Google index:
But if the page is not yet indexed, you can ask for an indexing:
Note : Requesting crawling does not guarantee that the URL will automatically be stored in Google’s index. In some cases, the indexation may not even take place. The case of a content of bad quality for example
2.5.2. Ask the indexation of a large number of URLs with the sitemap
The sitemap is a file in which Google gets an idea of the URLs that your site has. If you do not know how to create a sitemap, you can consult this article proposed by Google
If a sitemap has not been modified since the last exploration of Google, it is useless to send it again to the search engine. But if you have added pages to your sitemap, take care to tag it with the attribute
Here are the different steps to follow to alert Google of changes made to the sitemap
- Send a sitemap thanks to the sitemap report
- With your browser or the command editor, you can send a GET request to the following address by informing the complete link of the sitemap : http://www.google.com/ping?sitemap=
Example:
- Then, go to your Robots.txt file and add the following code
Sitemap: http://example.com/my_sitemap.xml
With the following steps, Google will be able to crawl the pages added to the sitemap the next time it visits
Chapter 3: Other questions about indexing
3.1) What does indexed by Google mean?
A page is said to be indexed by Google when it :
- Has been visited by the Google crawler (“called Googlebot”) ;
- Is the subject of a complete exploration
- Finally, stored in the Google index.
While the vast majority of web pages go through the crawling stage before indexing, Google may also index pages without accessing their content. For example, when a page is blocked by a robots.txt file.
3.2. What is web crawling and indexing?
Crawling: This is a stage of the Internet journey by crawlers in search of content, exploring the code/content of each URL they discover
Indexing: This is the step of storing and organizing the content found during the crawling process. Once a page is stored in the index, which is the large database, it is waiting to be displayed in response to relevant queries.
3.3) Why is web indexing useful?
Search engines index websites in order to answer search queries with relevant information as quickly as possible. For this reason, they store information about the indexed web pages, for example: keyword, title or description in a database. This way, search engines will be able to easily and quickly identify the relevant pages for a search query.
3.4. What are indexing errors?
If you get the message “the URL is not on google: indexing errors”, it means that Google has either removed the URL from its index because it could not access it, or that it was not in its index because it was not available during its first attempt.
3.5. Why is my site not on Google?
If your site does not appear on Google, it is probably for one of the following reasons:
- Google has not yet indexed your website
- Google considers that your site is not “trustworthy” or “relevant” enough to be displayed for the keywords you want to rank for
- You have blocked the Googlebot crawler in your robots file.
To resolve this issue, run a live inspection, resolve any issues you may have and submit the page for indexing.
3.6. How do I stop Google from indexing my site?
To prevent one of your pages from appearing in Google search, simply include a noindex meta tag in the HTML code of the page or by returning a noindex header in the HTTP response.
3.7. How to check if your page is indexed by Google?
To see if search engines such as Google and Bing have indexed your site, enter : ” site:URL of your domain ”
The results show all the pages of your site that have been indexed and the current Meta tags recorded in the index of the search engine.
It is important to know that the exploration of your site by search engines can sometimes take some time.
3.8. How much of the Internet is indexed by Google in 2020?
Google has stored in its index about 35 trillion web pages on the Internet worldwide. While this is an amazing statistic, believe it or not, 35 trillion is just the tip of the iceberg. Google’s index only represents about 4% of the information that exists on the net.
Conclusion
In short, indexing is an essential step for the referencing of a web page and is part of a process ensured by the indexing robots
After having newly created a website or published a new page, fully optimized for SEO, the best way to help its page to be indexed would be to wait
Although for some reasons, one may sometimes be forced to use methods to indicate some recommendations to search engines
I hope that this definition on “Indexing” has been useful and will add to your knowledge in SEO
If you have other questions, don’t hesitate to ask me in the comments
Thanks and see you soon!
