If you are looking for an SEO tool that will allow you to scrape competitor websites, crawl pages and validate links in minutes and provide information about any errors or warnings on your site, Screaming Frog is the ultimate must-have. The company’s website crawler can be customized to your needs and has a robust set of features. It provides insights into the pages of your site that need optimization to rank better in search engines like Google. You can use it to find duplicate content, check robots.txt files, and even export data from XML Sitemaps! This article discusses how Screaming Frog can help with everything from link building to content discovery.
What is Screaming Frog?
“Screaming Frog SEO Spider” is an SEO developer tool created by the UK-based search marketing agency Screaming Frog. Their SEO Spider is a website crawler that improves onsite SEO by extracting data & auditing for common SEO issues.
How is Screaming Frog useful?
With this tool, you can:
- Find broken links
- Audit redirects
- Analyze Page titles & metadata
- Discover duplicate content
- Extract data with CSS path, XPath, or Regex
- Review bots and directives
- Generate XML sitemaps
- Integrate with Google Analytics, Google Search Console, and PageSpeed Insights APIs
- Visualize website structure / architecture
- Custom Extractions
- Scrape SERPs
Why use Screaming Frog?
The SEO spider mimics a GoogleBot web crawler, which gives you valuable insight into how Google Search sees your website.
These website scans give you insight into how to fix possible website errors while providing key areas to focus on and improve your search results.
Is Screaming Frog free to use?
Yes, Screaming Frog is free to use, up to 500 pages per scan. If you wish to scan more pages, you will need to upgrade to a paid plan.
Where to download Screaming Frog?
Here is where you can download Screaming Frog for free:
Where can I get a Screaming Frog License?
You can purchase a Screaming Frog License from the official website:
How do I start using Screaming Frog SEO Spider?
- First, you need to download the screaming frog tool
- Install the software on your computer
- Open the program
- Add your domain name you wish to scan in the top toolbar
- Wait for the scan to finish, and analyze your results.
- Optional: Customize your Audit settings for the most optimial reporting
I will be creating a more in-depth guide on how to get started. This guide is to give you a broad understanding of how this program functions.
Screaming Frog FAQs
Screaming Frog shows 403 forbidden; what should I do?
While a server-side issue typically causes the 403 Forbidden error, there are a few things that you can do to try to troubleshoot the problem. First, Can you view the page in a web browser, or does it return a similar error? If you can’t access it from your computer, the server could be having issues, and it is best to retry later. If the page can be viewed in a web browser on your computer, set Chrome as the user agent in Screaming Frog, you can do this by going to “Configuration > User-Agent.” The server denies Screaming Frog’s SEO Spider’s page request, possibly as protection/security against unknown user agents, and sees it as malicious.
How to clear the cache on Screaming Frog?
Screaming Frog does not have a cache. So you will need to clear the cache locally; this means you need to flush the cache on the website you are crawling. In extreme cases, you will need to flush varnish at a server level. Restart your computer if that doesn’t work.
Can a custom user agent be set in Screaming Frog?
Yes, to set your Screaming Frog User-Agent, in the top Navigation Bar, click on Configuration >> User-Agent >> User-Agent Configuration >> select a Custom User-Agents you want to use.
How to crawl a sitemap in Screaming Frog?
To crawl a sitemap in Screaming Frog, click on Configuration >> Spider >> Scroll down to the XML Sitemap section on the first tab >> Select Crawl These Sitemaps >> Enter the Sitemap URL that you want to be crawled.
Why is Screaming Frog only crawling Homepage?
Screaming Frog does not crawling all URLs; How to fix it?
If Screaming Frog does not crawl all URLs on a website, In Screaming Frog’s settings, click on Configuration in the top navigation, select Spider, On the Crawl tab, down in the Crawl Behavior options, select “Follow internal nofollow” links. Having this setting enabled will allow Screaming Frog to crawl all links and pages on a website.
Learn more about practical Screaming Frog uses.
- How To Analyze Link Position using Screaming Frog
- How to Scrape Google SERPs with Screaming Frog
- Learn about Screaming Frog Custom Extractions
- The Best Screaming Frog settings for most Site Audits
Published on: 2021-03-10
Updated on: 2022-05-12