Crawling, indexing and ranking: the meanings & differences

As an SEO specialist, I use these terms daily: crawling, indexing and ranking . Crawling refers to the process by which search engines collect the content of Web pages on the Internet. Indexing is storing this content in the search engine’s database. Ranking refers to positioning the Web page in a search engine’s search results for a particular query.

In this article, I’ll take you through the world of crawling, indexing and ranking and the ways it affects an entire SEO campaign.

What is crawling, indexing and ranking?

First, I’ll give a brief explanation for each section, then I’ll go into more detail.

  1. Crawling: During this process, Google downloads a page’s text, images and videos with automated “programs” (crawlers).
  2. Indexing: Google analyzes this data and stores it in Google (which is basically a large database).
  3. Ranking: When someone searches for something in Google, this searcher may come across this data.

What is crawling?

Google discovers and maintains a list of Web pages through “URL discovery,” where it tracks links or uses sitemaps. A program called Googlebot crawls the pages according to an algorithm, carefully managing speed to avoid overloading the websites.

However, not all pages found are crawled; some are blocked by the site owner or require logins. During crawling, Google renders the page and executes JavaScript to ensure that all relevant content is displayed.(1)

  1. Googlebot finds a page.
  2. Googlebot downloads the page and all its content.
  3. Googlebot puts the links to other pages in the crawl queue, ready to crawl through.

What is indexing?

This part is called indexing and involves analyzing/understanding text, important tags such as

and alt attributes, as well as images and videos.

During indexing, Google looks to see if a page is a duplicate or the “real” version of this piece of content. The canonical is the page displayed in search results. Google first groups similar pages and selects the most representative one. Other pages in the group may be shown in specific situations, such as for mobile users. (2)

The process works as follows:

  1. Googlebot forms an image from the downloaded text, images, HTML and CSS.
  2. Googlebot executes the JavaScript found (this is a separate process), this becomes rendered HTML.
  3. GoogleBot forms a complete picture of the page.
  4. Googlebot indexes the page (when possible and Google itself wants it to).

What is tendril?

Now for the most fun part for many website owners; ranking. When a user enters a search query, Google’s systems search the index for matching pages and display the results they believe have the highest quality and relevance to the user.

Relevance is determined by hundreds of factors, such as location, language and device (desktop or phone). For example, the search “baker” will yield different results for users in Paris and Hong Kong.

The search results page also changes based on the search query. A search for “baker” is more likely to show local results, while “how to become a baker” is more likely to show national results.(3)

In brief:

  1. From the “processing” section, Google makes a decision (yes/no) to index a page.
  2. Then it is indexed and this page can rank in the index.

How do I affect crawling, indexing and ranking with SEO?

Now that you know what each part of crawling, indexing and ranking entails I’ll show you what I do within an SEO strategy to promote these parts.

What affects crawling and indexing by Googlebot?

So this is focused on how easily Google can access everything on the website. Some components that affect this:

  • Web site structure with a silo structure;
  • internal links to key pages;(4)
  • XML and HTML sitemaps;(5)
  • Optimize robots.txt (make sure Google can access everything);(6)
  • optimize canonicals;
  • Avoid noindex tags on important pages;
  • Avoid 4xx and 5xx pages;
  • Avoid redirect chains and/or unnecessary redirects;
  • Optimize the website for crawl budget.

How do I influence ranking?

This is beginning to look more like a regular SEO process, but parts that are important here, as follows:

  • website speed;
  • website structure;
  • quality content (and written on the right topics);
  • quality internal links;
  • good mobile usability (responsive);
  • a good user experience;
  • much value to potential visitors.

In short, all known things are important for this. For this purpose, I created the SEO roadmap.

Conclusion

In this article I explained what crawling, indexing and ranking mean and and how they differ from each other. A “real” SEO specialist knows the difference … now you do too. Good luck with optimization!

Resources

  1. In-Depth Guide to How Google Search Works | Google Search Central | Documentation | Google for Developers. (s.d.-j). Google For Developers. https://developers.google.com/search/docs/fundamentals/how-search-works#crawling
  2. In-Depth Guide to How Google Search Works | Google Search Central | Documentation | Google for Developers. (s.d.-k). Google For Developers. https://developers.google.com/search/docs/fundamentals/how-search-works#indexing
  3. In-Depth Guide to How Google Search Works | Google Search Central | Documentation | Google for Developers. (s.d.-l). Google For Developers. https://developers.google.com/search/docs/fundamentals/how-search-works#serving
  4. SEO Link Best Practices for Google | Google Search Central | Documentation | Google for Developers. (s.d.-g). Google For Developers. https://developers.google.com/search/docs/crawling-indexing/links-crawlable#internal-links
  5. What Is a Sitemap | Google Search Central | Documentation | Google for Developers. (s.d.-c). Google For Developers. https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview
  6. Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers. (s.d.). Google For Developers. https://developers.google.com/search/docs/crawling-indexing/robots/intro
Senior SEO-specialist

Ralf van Veen

Senior SEO-specialist
Five stars
My clients give me a 5.0 on Google out of 75 reviews

I have been working for 10 years as an independent SEO specialist for companies (in the Netherlands and abroad) that want to rank higher in Google in a sustainable manner. During this period I have consulted A-brands, set up large-scale international SEO campaigns and coached global development teams in the field of search engine optimization.

With this broad experience within SEO, I have developed the SEO course and helped hundreds of companies with improved findability in Google in a sustainable and transparent way. For this you can consult my portfolio, references and collaborations.

This article was originally published on 15 September 2023. The last update of this article was on 15 May 2024. The content of this page was written and approved by Ralf van Veen. Learn more about the creation of my articles in my editorial guidelines.