It can be very frustrating when you’re trying to crawl a website and suddenly see the 403 Forbidden error message. What do you do when you encounter this error? How can you fix it? In this blog post, we’ll explain the 403 Forbidden error and provide some solutions to fix it. Stay tuned!
What is a 403 Forbidden Error?
The 403 Forbidden status code is a server-side error indicating that the server refuses to fulfill the request. This can be due to several reasons, including but not limited to: the request is being made for a resource that does not exist, the user does not have sufficient permissions to access the requested resource, or the server is configured to block requests from the user’s IP address.
Regardless of the reason, a 403 Forbidden error will prevent the user from accessing the requested resource. In some cases, the server may provide a redirect to another page or resource that can be accessed; in other cases, the user will need to contact the website owner or administrator to resolve the issue. Either way, a 403 Forbidden error frustrates any user who encounters it.
How to fix 403 forbidden status code error in Screaming Frog
The page doesn’t exsit
Plausible cause 1: The request is being made for a resource that does not exist
You will need to verify that the web page exists. You can do this by opening the URL in your browser.
This test will show you if the web page is reachable from the browser.
Plausible cause 2: The user does not have sufficient permissions to access the requested resource.
In some cases, the page is password protected. You will need to end the username and password into the Screaming Frog modal popup for it to crawl that page.
The Server is blocking you
Plausible cause 3: The server is configured to block requests from the user’s IP address.
If the webpage is viewable on your computer using a browser, this means the website is blocking the bot (crawler) and is an isolated event.
Lower Crawl Speed
Solution 1: You might have the spider setup to crawl too fast and too aggressively.
To fix aggressive scans, you will need to lower the crawl rate:
Configuration >> Speed >> Spider Speed Configuration
Lowering your crawl rate will increase the time it takes to perform scans and reduce the server resources load.
Change the User-Agent
Solution 2: If the website is blocking the Screaming Frog Crawler, change the User-Agent in Screaming Frog.
Configuration >> User-Agent >> Present User-Agents
Some servers are set up to block anything except for a GoogleBot.
Change your IP
Solution 3: Change your IP address
The server could be blocking your IP address, so trying the crawl from a different internet service provider is best.
Try the crawl from home if you are at work and want to change your IP address. If you are at home, try a coffee shop with good internet. If you can’t leave, try tethering your computer to your phone. Changing locations will help you troubleshoot and determine if your IP is blocked.
Whitelist your IP
Solution 4: Whitelist your IP address on the Server or Website
You can whitelist your IP address with the website, server, or DNS server where possible.
The options are endless, but a few security services blocking your crawl could be: Wordfence, CloudFlare, and Fail2Ban; if you are unaware of any security software running, it’s best to consult with your IT department.
Don’t hesitate to contact us if you still have trouble fixing the 403 Forbidden error. We’ll be more than happy to help out!
How many people can use Screaming Frog license?
Licences are per user. 2 users will require 2 individual licences.
Published on: 2022-09-01
Updated on: 2022-09-27