Fixing Non-indexable URLs in Screaming Frog

Isaac Adams-Hands · Fixing Non-indexable URLs in Screaming Frog

Fixing non-indexable URLs in Screaming Frog can be a real pain. Fortunately, there is an easy way to do it. This blog post will show you how to fix them in just a few minutes. So, if you’re having trouble getting your URLs indexed, read on! We’ll show you how to fix them in Screaming Frog and get your website ranking higher in Google search results.

How to fix Non-indexable URLs in Screaming Frog

  1. Indexability Status Codes

    Indexability Status Codes

    Four principal indexability status codes exist: 2XX, 3XX, 4XX, and 5XX.

    200 code: a search engine can successfully access the page.
    301 code: the page has been permanently moved to a new location.
    302 code: the page has been temporarily moved to a new location.
    404 code: the page could not be found.
    500 code: the server encountered a client error.

    These codes are essential for understanding how a search engine interacts with your website.

    Understanding these indexability status codes allows you to control better how screaming frog encounters an error when scanning your website.

  2. 200 Status Codes

    discourage search engines

    In most cases, a no-index tag has been applied to the website creating a Non-indexable status.

    If you have a WordPress website, you can check Settings >> Reading >> Search Engine Visibility to see if the box has been checked causing the website to be set as no-index.

    If that doesn’t resolve your issue, you can check individual pages for a no-index tag; if you are using the Yoast plugin, you can check at the page level under Advanced >> Allow search engines to show this Page in search results?

    allow search engines

    You can also check your Robots.TXT file, which can be found on your website by going to the domain name followed by domain.com/robots.txt. You will be looking for a Disallow: / setting.

    Finally, you can look at your page’s canonical tags

    canonical tag

    The URL in the browser and rel="canonical tag" should match. If they are incorrect (e.g., Rel is set to HTTP and the webpage is set to HTTPS), you can correct the canonical URL using the Better Search ‘n Replace Plugin or in the Yoast plugin.

  3. 3XX Error Codes

    When you see a 3XX error code, the URL is redirected. You will need to manually fix the URL redirects on your website or use a plugin. 3XX errors mean that search engines can only index the final destination.

  4. 404 Error Codes

    redirection plugin

    If you are experiencing 4XX errors, you can fix the links manually or use a plugin like Redirection. 404 errors are not indexable as they are the “Page Not Found” warning page.

    403 errors indicates in most cases that an external website is blocking your crawler for security and safety reasons.

  5. 5XX Error Codes

    Internal Server Error

    5XX error code means an Internal Server Error occurred. This error is commonly caused by Screaming Frog scanning too fast for the server to keep up and becoming overloaded, running out of server resources (memory).

    To fix this error, you can upgrade your website’s server or slow down how fast the crawler scans your website.

    In Screaming Frog, you can update the crawl speed by going to Configuration >> Speed >> Max Thread.

    speed

Conclusion

By following our guide on fixing non-indexable pages or non-indexable canonical tags in Screaming Frog, you can ensure that as many people see your website as possible. This will help improve your site’s ranking on Google search results pages and increase the chances that potential customers will find your business online. For more information or tutorials, you can visit the Screaming Frog website or get an SEO expert who would do these technical SEO for you.

FAQ

  • How can you identify and fix non-indexable URLs in Screaming Frog SEO Spider?

Published on: 2022-09-26
Updated on: 2024-05-07

Avatar for Isaac Adams-Hands

Isaac Adams-Hands

Isaac Adams-Hands is the SEO Director at SEO North, a company that provides Search Engine Optimization services. As an SEO Professional, Isaac has considerable expertise in On-page SEO, Off-page SEO, and Technical SEO, which gives him a leg up against the competition.
en_CAEnglish