This blog post will discuss how to use screaming frog to crawl a subfolder only. We’ll be walking through the process of setting up Screaming Frog and configuring it to crawl a specific folder on your website. This is an excellent tool for auditing specific website sections or troubleshooting issues with certain pages. Let’s get started!
Table of Contents
How do you crawl a subfolder in Screaming Frog?
Spider Mode
Set Screaming Frog to Spider mode and enter the website URL you wish to crawl.
Set Includes
In the navigation menu at the top, go to Configure >> then click on Includes in the dropdown.
Enter the subfolder you wish to crawl, followed by a period and star enabling Regex to only to crawl that folder.
e.g.,
https://seonorth.ca/seo/.*If the URL is correct, you will see a checkmark symbol towards the bottom of the modal.
View Results
Press start in Screaming frog and watch it crawl the website; you can monitor the crawl to see that it is only crawling that folder.
Alternatively, you can set Screaming Frog to Exclude specific subfolders by going to Configure >> Exclude in the top Navigation.
Conclusion
Now that you know how to crawl a subfolder, put this knowledge to use for your website. Screaming Frog is essential for any SEOs toolkit and can help you uncover critical data about your website. Use the information you gather from Screaming Frog to improve your website’s performance and boost your search engine rankings. What are some of the other ways you’ve found it helpful? Let us know in the comments below.
FAQ
How do you crawl a specific folder in Screaming Frog?
Published on: 2022-09-26
Updated on: 2022-10-28
