How to only crawl a subfolder using Screaming Frog

screaming frog
How to set up Screaming Frog only to crawl a subfolder.
Isaac Adams-Hands · How to only crawl a subfolder using Screaming Frog

This blog post will discuss how to use screaming frog to crawl a subfolder only. We’ll be walking through the process of setting up Screaming Frog and configuring it to crawl a specific folder on your website. This is an excellent tool for auditing specific website sections or troubleshooting issues with certain pages. Let’s get started!

How do you crawl a subfolder in Screaming Frog?

  1. Spider Mode

    spider mode

    Set Screaming Frog to Spider mode and enter the website URL you wish to crawl.

  2. Set Includes

    includes

    In the navigation menu at the top, go to Configure >> then click on Includes in the dropdown.

    Enter the subfolder you wish to crawl, followed by a period and star enabling Regex to only to crawl that folder.

    e.g., https://seonorth.ca/seo/.*

    If the URL is correct, you will see a checkmark symbol towards the bottom of the modal.

  3. View Results

    results

    Press start in Screaming frog and watch it crawl the website; you can monitor the crawl to see that it is only crawling that folder.

    Alternatively, you can set Screaming Frog to Exclude specific subfolders by going to Configure >> Exclude in the top Navigation.

Conclusion

Now that you know how to crawl a subfolder, put this knowledge to use for your website. Screaming Frog is essential for any SEOs toolkit and can help you uncover critical data about your website. Use the information you gather from Screaming Frog to improve your website’s performance and boost your search engine rankings. What are some of the other ways you’ve found it helpful? Let us know in the comments below.

FAQ

  • How do you crawl a specific folder in Screaming Frog?

Published on: 2022-09-26
Updated on: 2024-03-20

Avatar for Isaac Adams-Hands

Isaac Adams-Hands

Isaac Adams-Hands is the SEO Director at SEO North, a company that provides Search Engine Optimization services. As an SEO Professional, Isaac has considerable expertise in On-page SEO, Off-page SEO, and Technical SEO, which gives him a leg up against the competition.
en_CAEnglish