How to only crawl a subfolder using Screaming Frog

screaming frog
How to set up Screaming Frog only to crawl a subfolder.
Isaac Adams-Hands · How to only crawl a subfolder using Screaming Frog

This blog post will discuss how to use screaming frog to crawl a subfolder only. We’ll be walking through the process of setting up Screaming Frog and configuring it to crawl a specific folder on your website. This is an excellent tool for auditing specific website sections or troubleshooting issues with certain pages. Let’s get started!

How do you crawl a subfolder in Screaming Frog?

  1. Spider Mode

    spider mode

    Set Screaming Frog to Spider mode and enter the website URL you wish to crawl.

  2. Set Includes


    In the navigation menu at the top, go to Configure >> then click on Includes in the dropdown.

    Enter the subfolder you wish to crawl, followed by a period and star enabling Regex to only to crawl that folder.


    If the URL is correct, you will see a checkmark symbol towards the bottom of the modal.

  3. View Results


    Press start in Screaming frog and watch it crawl the website; you can monitor the crawl to see that it is only crawling that folder.

    Alternatively, you can set Screaming Frog to Exclude specific subfolders by going to Configure >> Exclude in the top Navigation.


Now that you know how to crawl a subfolder put this knowledge to use for your website. Screaming Frog is essential for any SEO toolkit and can help you uncover critical data about your website. Use the information you gather from Screaming Frog to improve your website’s performance and boost your search engine rankings. What are some of the other ways you’ve found it helpful? Let us know in the comments below.


  • What is Screaming Frog?
  • How does Screaming Frog handle JavaScript?
  • Can Screaming Frog detect broken links?
  • What is internal linking, and how does Screaming Frog help with it?
  • Does Screaming Frog support crawling subdomains?
  • Where can I find tutorials for using Screaming Frog?
  • What is a canonical URL, and does Screaming Frog identify them?
  • Can Screaming Frog crawl HTML and CSS content?
  • How do I ensure that my website is indexable using Screaming Frog?
  • Does Screaming Frog integrate with other tools like Ahrefs or Google Analytics?
  • Can I use the Screaming Frog API to automate tasks?
  • How can I crawl outside of the start folder using Screaming Frog?
  • Does Screaming Frog support custom extraction of data?
  • How does Screaming Frog help identify duplicate content?
  • Can Screaming Frog analyze Google Analytics data?
  • How does Screaming Frog handle JavaScript rendering for SEO analysis?
  • Is there a limit to the number of URLs that Screaming Frog can crawl?
  • How does Screaming Frog handle response codes during crawling?
  • Does Screaming Frog respect robots.txt directives?
  • How do you crawl a specific folder in Screaming Frog?

Published on: 2022-09-26
Updated on: 2024-04-05

Avatar for Isaac Adams-Hands

Isaac Adams-Hands

Isaac Adams-Hands is the SEO Director at SEO North, a company that provides Search Engine Optimization services. As an SEO Professional, Isaac has considerable expertise in On-page SEO, Off-page SEO, and Technical SEO, which gives him a leg up against the competition.