Crawling, indexing and ranking: the meanings & differences

As an SEO specialist, I use these terms daily: crawling, indexing and ranking . Crawling refers to the process by which search engines collect the content of Web pages on the Internet. Indexing is storing this content in the search engine’s database. Ranking refers to positioning the Web page in a search engine’s search results for a particular query.

In this article, I’ll take you through the world of crawling, indexing and ranking and the ways it affects an entire SEO campaign.

What is crawling, indexing and ranking?

First, I’ll give a brief explanation for each section, then I’ll go into more detail.

1. Crawling: During this process, Google downloads a page’s text, images and videos with automated “programs” (crawlers).
2. Indexing: Google analyzes this data and stores it in Google (which is basically a large database).
3. Ranking: When someone searches for something in Google, this searcher may come across this data.

What is crawling?

Google discovers and maintains a list of Web pages through “URL discovery,” where it tracks links or uses sitemaps. A program called Googlebot crawls the pages according to an algorithm, carefully managing speed to avoid overloading the websites.

However, not all pages found are crawled; some are blocked by the site owner or require logins. During crawling, Google renders the page and executes JavaScript to ensure that all relevant content is displayed.(1)

1. Googlebot finds a page.
2. Googlebot downloads the page and all its content.
3. Googlebot puts the links to other pages in the crawl queue, ready to crawl through.

What is indexing?

This part is called indexing and involves analyzing/understanding text, important tags such as

During indexing, Google looks to see if a page is a duplicate or the “real” version of this piece of content. The canonical is the page displayed in search results. Google first groups similar pages and selects the most representative one. Other pages in the group may be shown in specific situations, such as for mobile users. (2)

The process works as follows:

1. Googlebot forms an image from the downloaded text, images, HTML and CSS.
2. Googlebot executes the JavaScript found (this is a separate process), this becomes rendered HTML.
3. GoogleBot forms a complete picture of the page.
4. Googlebot indexes the page (when possible and Google itself wants it to).

What is tendril?

Now for the most fun part for many website owners; ranking. When a user enters a search query, Google’s systems search the index for matching pages and display the results they believe have the highest quality and relevance to the user.

Relevance is determined by hundreds of factors, such as location, language and device (desktop or phone). For example, the search “baker” will yield different results for users in Paris and Hong Kong.

The search results page also changes based on the search query. A search for “baker” is more likely to show local results, while “how to become a baker” is more likely to show national results.(3)

In brief:

1. From the “processing” section, Google makes a decision (yes/no) to index a page.
2. Then it is indexed and this page can rank in the index.

How do I affect crawling, indexing and ranking with SEO?

Now that you know what each part of crawling, indexing and ranking entails I’ll show you what I do within an SEO strategy to promote these parts.

What affects crawling and indexing by Googlebot?

So this is focused on how easily Google can access everything on the website. Some components that affect this:

• Web site structure with a silo structure;
• internal links to key pages;(4)
• XML and HTML sitemaps;(5)
• Optimize robots.txt (make sure Google can access everything);(6)
• optimize canonicals;
• Avoid noindex tags on important pages;
• Avoid 4xx and 5xx pages;
• Avoid redirect chains and/or unnecessary redirects;
• Optimize the website for crawl budget.

How do I influence ranking?

This is beginning to look more like a regular SEO process, but parts that are important here, as follows:

• website speed;
• website structure;
• quality content (and written on the right topics);
• quality internal links;
• good mobile usability (responsive);
• a good user experience;
• much value to potential visitors.

In short, all known things are important for this. For this purpose, I created the SEO roadmap.

Conclusion

In this article I explained what crawling, indexing and ranking mean and and how they differ from each other. A “real” SEO specialist knows the difference … now you do too. Good luck with optimization!