Fixing non-indexable URLs in Screaming Frog can be a real pain. Fortunately, there is an easy way to do it. This blog post will show you how to fix them in just a few minutes. So, if you’re having trouble getting your URLs indexed, read on! We’ll show you how to fix them in Screaming Frog and get your website ranking higher in Google search results.
Table of Contents
How to fix Non-indexable URLs in Screaming Frog
Indexability Status Codes
Four principal indexability status codes exist: 2XX, 3XX, 4XX, and 5XX.
200 code: a search engine can successfully access the page.
301 code: the page has been permanently moved to a new location.
302 code: the page has been temporarily moved to a new location.
404 code: the page could not be found.
500 code: the server encountered an error.
These codes are essential for understanding how a search engine interacts with your website.
Understanding these indexability status codes allows you to control better how screaming frog encounters an error when scanning your website.
200 Status Codes
In most cases, a no-index tag has been applied to the website creating a Non-indexable status.
If you have a WordPress website, you can check Settings >> Reading >> Search Engine Visibility to see if the box has been checked causing the website to be set as no-index.
If that doesn’t resolve your issue, you can check individual pages for a no-index tag; if you are using the Yoast plugin, you can check at the page level under Advanced >> Allow search engines to show this Page in search results?
You can also check your Robots.TXT file, which can be found on your website by going to the domain name followed by domain.com/robots.txt. You will be looking for a
Finally, you can look at your page’s canonical tags
The URL in the browser and
rel="canonical tag"should match. If they are incorrect (e.g., Rel is set to HTTP and the webpage is set to HTTPS), you can correct the URL using the Better Search ‘n Replace Plugin.
3XX Error Codes
When you see a 3XX error code, the URL is redirected. You will need to manually fix the URL redirects on your website or use a plugin. 3XX errors mean that search engines can only index the final destination.
404 Error Codes
If you are experiencing 4XX errors, you can fix the links manually or use a plugin like Redirection. 404 errors are not indexable as they are the “Page Not Found” warning page.
403 errors indicates in most cases that an external website is blocking your crawler for security and safety reasons.
5XX Error Codes
5XX error code means an Internal Server Error occurred. This error is commonly caused by Screaming Frog scanning too fast for the server to keep up and becoming overloaded, running out of server resources (memory).
To fix this error, you can upgrade your website’s server or slow down how fast the crawler scans your website.
In Screaming Frog, you can update the crawl speed by going to Configuration >> Speed >> Max Thread.
By following our guide on fixing non-indexable pages in Screaming Frog, you can ensure that as many people see your website as possible. This will help improve your site’s ranking on Google search results pages and increase the chances that potential customers will find your business online.
What does non-indexable mean in Screaming Frog?
Screaming Frog is a website crawler that allows you to quickly and easily analyze your website’s structure and on-page elements. One of the key metrics that it tracks is the “indexability” of each page. This simply refers to whether or not a page can be found and indexed by search engine crawlers. If a page is non-indexable, it means that it might not be indexed by search engines and most likely not appear in any search results. There are a number of reasons why a page might be non-indexable, but the most common reason is that the page has been blocked by a no-index tag. In other words, the website owner has specifically told search engines not to crawl that particular page. Non-indexable pages can also sometimes be caused by technical issues, such as incorrect redirects or 404 errors. Regardless of the cause, pages that are non-indexable will not contribute to your website’s SEO and should be fixed as soon as possible.
What does indexable mean in Screaming Frog?
Indexable means that a search engine can read and understand the content on a given page. This is important because it allows the search engine to determine whether or not the page is relevant to a given search query. If a page is not indexable, it will not show up in the search results. There are a number of factors that can make a page unindexable, such as using invalid HTML code or including blocked content. However, the most common reason for a page to be unindexable is simply because the search engine has not yet crawled it. In order to ensure that your pages are indexed, it is important to submit a sitemap to the major search engines. This will give them a list of all of your pages, and help them to crawl your site more effectively.
Published on: 2022-09-26
Updated on: 2022-10-28