How can you identify and fix non-indexable URLs in Screaming Frog SEO Spider?

To fix non-indexable URLs using Screaming Frog SEO Spider, you should first crawl your site to identify these URLs. Look at the 'Indexability Status’ column to see which pages are non-indexable. Factors causing non-indexability can include duplicate content, no response (404 errors), or meta tags like 'nofollow'. To resolve these issues, check the following: HTML and JavaScript Rendering: Ensure that Screaming Frog's user-agent settings match those of Google's bots to accurately simulate how your site is crawled. Enable JavaScript rendering if your site relies on it for content generation. Canonical Link Elements: Verify that canonical tags are correctly implemented. Non-indexable pages due to incorrect canonicalisation should have the canonical link element pointing to the correct, indexable URL. Meta and Header Configuration: Review the meta tags and header responses of non-indexable pages. Remove 'nofollow' directives and correct any misconfigured 'noindex' tags. Sitemap and Crawl Depth: Make sure your XML sitemap is updated and includes all indexable URLs. In Screaming Frog, analyze the crawl depth to ensure important pages aren't buried too deep for search engines to easily find. Duplicate Content and Alt Text: Address duplicate content issues and ensure that alt text is used correctly for images, which can affect indexability. Validation and Config in Google Search Console: After making changes, validate them through Google Search Console. This will give you insight into how Google views your site post-correction. Right-hand Details Panel in Screaming Frog: Use the right-hand details panel in Screaming Frog to inspect individual URLs for specific errors or warnings related to indexability. By addressing these elements, you can enhance the indexability of your site's URLs, improving their visibility in Google search results.

Fixing Non-indexable URLs in Screaming Frog

Isaac Adams-Hands · Fixing Non-indexable URLs in Screaming Frog

Fixing non-indexable URLs in Screaming Frog can be a real pain. Fortunately, there is an easy way to do it. This blog post will show you how to fix them in just a few minutes. So, if you’re having trouble getting your URLs indexed, read on! We’ll show you how to fix them in Screaming Frog and get your website ranking higher in Google search results.

Table of Contents

How to fix Non-indexable URLs in Screaming Frog
Indexability Status Codes
200 Status Codes
3XX Error Codes
404 Error Codes
5XX Error Codes
Conclusion
FAQ

How to fix Non-indexable URLs in Screaming Frog

Indexability Status Codes
Four principal indexability status codes exist: 2XX, 3XX, 4XX, and 5XX.

200 code: a search engine can successfully access the page.
301 code: the page has been permanently moved to a new location.
302 code: the page has been temporarily moved to a new location.
404 code: the page could not be found.
500 code: the server encountered a client error.

These codes are essential for understanding how a search engine interacts with your website.

Understanding these indexability status codes allows you to control better how screaming frog encounters an error when scanning your website.
200 Status Codes
In most cases, a no-index tag has been applied to the website creating a Non-indexable status.

If you have a WordPress website, you can check Settings >> Reading >> Search Engine Visibility to see if the box has been checked causing the website to be set as no-index.

If that doesn’t resolve your issue, you can check individual pages for a no-index tag; if you are using the Yoast plugin, you can check at the page level under Advanced >> Allow search engines to show this Page in search results?

You can also check your Robots.TXT file, which can be found on your website by going to the domain name followed by domain.com/robots.txt. You will be looking for a Disallow: / setting.

Finally, you can look at your page’s canonical tags

The URL in the browser and rel="canonical tag" should match. If they are incorrect (e.g., Rel is set to HTTP and the webpage is set to HTTPS), you can correct the canonical URL using the Better Search ‘n Replace Plugin or in the Yoast plugin.
3XX Error Codes
When you see a 3XX error code, the URL is redirected. You will need to manually fix the URL redirects on your website or use a plugin. 3XX errors mean that search engines can only index the final destination.
404 Error Codes
If you are experiencing 4XX errors, you can fix the links manually or use a plugin like Redirection. 404 errors are not indexable as they are the “Page Not Found” warning page.

403 errors indicates in most cases that an external website is blocking your crawler for security and safety reasons.
5XX Error Codes
5XX error code means an Internal Server Error occurred. This error is commonly caused by Screaming Frog scanning too fast for the server to keep up and becoming overloaded, running out of server resources (memory).

To fix this error, you can upgrade your website’s server or slow down how fast the crawler scans your website.

In Screaming Frog, you can update the crawl speed by going to Configuration >> Speed >> Max Thread.

Conclusion

By following our guide on fixing non-indexable pages or non-indexable canonical tags in Screaming Frog, you can ensure that as many people see your website as possible. This will help improve your site’s ranking on Google search results pages and increase the chances that potential customers will find your business online. For more information or tutorials, you can visit the Screaming Frog website or get an SEO expert who would do these technical SEO for you.

FAQ

How can you identify and fix non-indexable URLs in Screaming Frog SEO Spider?
To fix non-indexable URLs using Screaming Frog SEO Spider, you should first crawl your site to identify these URLs. Look at the ‘Indexability Status’ column to see which pages are non-indexable. Factors causing non-indexability can include duplicate content, no response (404 errors), or meta tags like ‘nofollow’. To resolve these issues, check the following:
1. HTML and JavaScript Rendering: Ensure that Screaming Frog’s user-agent settings match those of Google’s bots to accurately simulate how your site is crawled. Enable JavaScript rendering if your site relies on it for content generation.
2. Canonical Link Elements: Verify that canonical tags are correctly implemented. Non-indexable pages due to incorrect canonicalisation should have the canonical link element pointing to the correct, indexable URL.
3. Meta and Header Configuration: Review the meta tags and header responses of non-indexable pages. Remove ‘nofollow’ directives and correct any misconfigured ‘noindex’ tags.
4. Sitemap and Crawl Depth: Make sure your XML sitemap is updated and includes all indexable URLs. In Screaming Frog, analyze the crawl depth to ensure important pages aren’t buried too deep for search engines to easily find.
5. Duplicate Content and Alt Text: Address duplicate content issues and ensure that alt text is used correctly for images, which can affect indexability.
6. Validation and Config in Google Search Console: After making changes, validate them through Google Search Console. This will give you insight into how Google views your site post-correction.
7. Right-hand Details Panel in Screaming Frog: Use the right-hand details panel in Screaming Frog to inspect individual URLs for specific errors or warnings related to indexability.
By addressing these elements, you can enhance the indexability of your site’s URLs, improving their visibility in Google search results.

Published on: 2022-09-26
Updated on: 2024-05-07

Isaac Adams-Hands

Isaac Adams-Hands is the SEO Director at SEO North, a company that provides Search Engine Optimization services. As an SEO Professional, Isaac has considerable expertise in On-page SEO, Off-page SEO, and Technical SEO, which gives him a leg up against the competition.