This blog post will discuss how to use screaming frog to crawl a subfolder only. We’ll be walking through the process of setting up Screaming Frog and configuring it to crawl a specific folder on your website. This is an excellent tool for auditing specific website sections or troubleshooting issues with certain pages. Let’s get started!
Table of Contents
How do you crawl a subfolder in Screaming Frog?
Set Screaming Frog to Spider mode and enter the website URL you wish to crawl.
In the navigation menu at the top, go to Configure >> then click on Includes in the dropdown.
Enter the subfolder you wish to crawl, followed by a period and star enabling Regex to only to crawl that folder.
If the URL is correct, you will see a checkmark symbol towards the bottom of the modal.
Press start in Screaming frog and watch it crawl the website; you can monitor the crawl to see that it is only crawling that folder.
Alternatively, you can set Screaming Frog to Exclude specific subfolders by going to Configure >> Exclude in the top Navigation.
Now that you know how to crawl a subfolder, put this knowledge to use for your website. Screaming Frog is essential for any SEOs toolkit and can help you uncover critical data about your website. Use the information you gather from Screaming Frog to improve your website’s performance and boost your search engine rankings. What are some of the other ways you’ve found it helpful? Let us know in the comments below.
How do you crawl a specific folder in Screaming Frog?
In the navigation menu at the top of Screaming Frog, go to Configure >> then click on “Includes” in the dropdown. Enter the subfolder URL you want to crawl.
Published on: 2022-09-26
Updated on: 2022-10-28