ScreamingFrog

Looking for a free tool to crawl websites? Take ScreamingFrog, for example. This is 1 of the possibilities for a crawler. There is a limit to the free version, though. You can crawl a maximum of 500 URLs, in fact. After that, you will have to pay. There is only 1 paid version.

What is ScreamingFrog?

Screaming Frog is thus a crawler. That is, an automated software that searches the Internet for sites and pages. A crawler is also called spiderbot or search bot. Screaming Frog is a tool you can download and install on your computer. It is available for several operating systems.

Features of ScreamingFrog

But what exactly does Screaming Frog do? There are several functions of the program. For example, finding broken links, such as links that lead to a page with a 404 error. It is also possible to do an audit of link redirects with Screaming Frog. With an algorithm check, Screaming Frog finds any duplicate content or URLs. It is possible to see the URLs that are blocked by robots.txt or meta robots, for example. The paid version gives the option to schedule audits. With advanced URL mapping, you can compare previous crawls and current crawls.

Impact on SEO

That difference can provide insight into how your SEO has changed. Is your site in better shape or worse? Screaming Frog gives you numerous insights that can contribute to your SEO. In fact, search engines also use crawlers to determine the position of search results for their listings. By doing the same yourself, you provide insight into it.

My advice

This is a nice crawler tool with many features. With 500 URLs, you can lean on the free version for a while. That way you get to know the program before you pay for it. However, the paid version does have some additional features, including scheduling crawls. Often crawlers are extensions that take up space in your browser. Screaming Frog again is not.