Robots.txt is a file found in a website’s root folder, or root. A robots.txt has several functions, such as telling search engine crawlers which Web site or Web page they may or may not search. This is also important in getting higher in Google.

What is the robots.txt?

A robots.txt file is a text file that is publicly accessible. It is a very important file because it tells a search engine exactly what part of a website a search engine crawler may and may not index. This can significantly affect the performance of the website.

It is for search engine optimization(an SEO campaign, for example) a very powerful and also technical tool. Is there no robots.txt placed on the Web server? Then a search engine can search the entire website.

Examples of robots.txt

It is very important to note that a robots.txt does not remove a page that is already indexed. It will only prevent a page from being indexed by a search engine’s crawlers. Thus, if a page has been online for a while before a robots.txt file was added, it will still remain stored in the search engine’s database.

The importance of robots.txt

When a search engine crawler visits a particular website, it will always check the robots.txt file first. Actually, robots.txt is part of the Robots Exclusion Protocol (REP). This protocol regulates how robots are allowed to index and crawl the Web. This is useful if, for example, you don’t want a search engine to index duplicate pages.

The impact of robots.txt

The robots.txt file has been around since 1994 and was designed by Dutchman Martijn Koster. His website was put in the bad light by a search engine. Therefore, the purpose of robots.txt is to gain more control over your ranking and SEO. So very important if you want your website to rank well.

My advice

For anyone working with a website SEO consultant, it is helpful to know exactly what robots.txt means and what it is all used for. If you have not placed robots.txt on your website, the search engines will crawl the entire website, which you may prefer to avoid.

Senior SEO-specialist

Ralf van Veen

Senior SEO-specialist
Five stars
My clients give me a 5.0 on Google out of 75 reviews

I have been working for 10 years as an independent SEO specialist for companies (in the Netherlands and abroad) that want to rank higher in Google in a sustainable manner. During this period I have consulted A-brands, set up large-scale international SEO campaigns and coached global development teams in the field of search engine optimization.

With this broad experience within SEO, I have developed the SEO course and helped hundreds of companies with improved findability in Google in a sustainable and transparent way. For this you can consult my portfolio, references and collaborations.

This article was originally published on 22 March 2024. The last update of this article was on 22 March 2024. The content of this page was written and approved by Ralf van Veen. Learn more about the creation of my articles in my editorial guidelines.