This blog post will discuss how to use screaming frog to crawl a subfolder only. We’ll be walking through the process of setting up Screaming Frog and configuring it to crawl a specific folder on your website. This is an excellent tool for auditing specific website sections or troubleshooting issues with certain pages. Let’s get started!
Set Screaming Frog to Spider mode and enter the website URL you wish to crawl.
Set Includes
In the navigation menu at the top, go to Configure >> then click on Includes in the dropdown.
Enter the subfolder you wish to crawl, followed by a period and star enabling Regex to only to crawl that folder.
e.g., https://seonorth.ca/seo/.*
If the URL is correct, you will see a checkmark symbol towards the bottom of the modal.
View Results
Press start in Screaming frog and watch it crawl the website; you can monitor the crawl to see that it is only crawling that folder.
Alternatively, you can set Screaming Frog to Exclude specific subfolders by going to Configure >> Exclude in the top Navigation.
Conclusion
Now that you know how to crawl a subfolder put this knowledge to use for your website. Screaming Frog is essential for any SEO toolkit and can help you uncover critical data about your website. Use the information you gather from Screaming Frog to improve your website’s performance and boost your search engine rankings. What are some of the other ways you’ve found it helpful? Let us know in the comments below.
FAQ
What is Screaming Frog?
Screaming Frog is a powerful website crawler tool used for analyzing and auditing websites for various SEO factors and technical issues.
How does Screaming Frog handle JavaScript?
Screaming Frog now renders JavaScript when crawling websites, allowing for a more comprehensive analysis of pages that rely on JavaScript for content or functionality.
Can Screaming Frog detect broken links?
Yes, Screaming Frog can identify broken links (404 errors), redirect chains, and other issues related to links on a website during the crawling process.
What is internal linking, and how does Screaming Frog help with it?
Internal linking refers to linking between pages within the same website. Screaming Frog’s “Internal” tab provides insights into the internal linking structure of a website, helping users understand how pages are interconnected.
Does Screaming Frog support crawling subdomains?
Yes, Screaming Frog can crawl subdomains as long as they are within the same root domain.
Where can I find tutorials for using Screaming Frog?
Screaming Frog offers comprehensive tutorials and guides on its official website, covering various features and functionalities of the tool.
What is a canonical URL, and does Screaming Frog identify them?
Canonical URLs are used to indicate the preferred version of a web page. Screaming Frog can detect canonical tags and help identify potential issues with canonicalization.
Can Screaming Frog crawl HTML and CSS content?
Yes, Screaming Frog can crawl and analyze both HTML and CSS content on web pages.
How do I ensure that my website is indexable using Screaming Frog?
Screaming Frog provides insights into indexability issues such as meta robots tags, noindex directives, and more, helping users ensure that their website is properly indexed by search engines. You can also change the User Agent to GoogleBot to see how well Google can craw your website.
Does Screaming Frog integrate with other tools like Ahrefs or Google Analytics?
While Screaming Frog doesn’t have direct integrations with tools like Ahrefs or Google Analytics, it provides various export options (e.g., CSV, Google Sheets) that allow users to combine data from multiple sources for comprehensive analysis.
Can I use the Screaming Frog API to automate tasks?
Yes, Screaming Frog offers an API that allows users to automate tasks, extract data, and integrate Screaming Frog with other tools or scripts.
How can I crawl outside of the start folder using Screaming Frog?
Screaming Frog allows users to specify custom crawl configurations, including crawling outside of the initial start folder, by adjusting the settings in the “Configuration” section.
Does Screaming Frog support custom extraction of data?
Yes, Screaming Frog offers a custom search and extraction features that allows users to perform data extraction on specific data elements from web pages using XPath, CSS Path, or Regex.
How does Screaming Frog help identify duplicate content?
Screaming Frog’s “Duplicate Content” feature identifies pages with similar or identical content, helping users address duplicate content issues that may negatively affect SEO.
Can Screaming Frog analyze Google Analytics data?
While Screaming Frog doesn’t directly analyze Google Analytics data, it can be used in conjunction with Google Analytics to correlate website analytics data with technical SEO insights gathered during crawling.
How does Screaming Frog handle JavaScript rendering for SEO analysis?
Screaming Frog’s JavaScript rendering capability allows it to render web pages as a modern browser would, enabling users to analyze JavaScript-generated content and its impact on SEO.
Is there a limit to the number of URLs that Screaming Frog can crawl?
The number of URLs that Screaming Frog can crawl depends on the license type (e.g., free, paid) and the resources available on the user’s machine. Paid versions of Screaming Frog offer higher URL limits and additional features.
How does Screaming Frog handle response codes during crawling?
Screaming Frog provides detailed information about HTTP response codes encountered during crawling, helping users identify issues such as 404 errors, redirects, and server errors.
Does Screaming Frog respect robots.txt directives?
Yes, Screaming Frog respects robots.txt directives by default, but users can adjust the settings to ignore robots.txt rules if necessary.
How do you crawl a specific folder in Screaming Frog?
To have the Screaming Frog SEO Spider crawl outside of start folder: In the navigation menu at the top of Screaming Frog, go to Configure >> then click on “Includes” in the dropdown. Enter the subfolder URL you want to crawl.
Isaac Adams-Hands is the SEO Director at SEO North, a company that provides Search Engine Optimization services. As an SEO Professional, Isaac has considerable expertise in On-page SEO, Off-page SEO, and Technical SEO, which gives him a leg up against the competition.
English 
Portuguese 
Japanese 
French 
Spanish 
Chinese