Screaming Frog acting sluggish? Throwing errors during your SEO crawls? The culprit is often its cache. This guide will show you exactly how to clear Screaming Frog’s cache, reclaim disk space, and get back to analyzing web pages efficiently.
Table of Contents
Why Screaming Frog’s Cache Matters
When Screaming Frog crawls a website, it stores vast amounts of data locally – status codes, page titles, meta tags, headers, inlinks, outlinks, and more. This cache functions like short-term memory, allowing the crawler to work faster by avoiding redundant requests to search engines and servers. However, this cached data can become problematic.
The cache stores everything from HTML source code and JavaScript rendering results to response codes and http headers. For large-scale crawls analyzing thousands of web pages, these files can consume gigabytes of disk space. More critically, corrupted or outdated cache data can cause the application to crash, display incorrect metrics, or contaminate new crawls with old information.
Common Reasons to Clear the Cache
Troubleshooting Errors and Crashes
If Screaming Frog freezes, crashes, or displays unexpected validation errors, a corrupted cache is typically the first suspect. Fragmented data can confuse the application, particularly when switching between projects or crawling sites with complex structured data and schema markup.
Freeing Up Disk Space
Enterprise-level crawls generate enormous project files. When you’re regularly crawling sites with extensive indexing requirements, xml sitemaps, and numerous subdomains, the cache directories can quickly balloon to tens of gigabytes. This is especially true if you’re using the API for bulk export operations or running multiple crawls of different homepages.
Ensuring Fresh Crawl Data
After major website updates, migrations, or when implementing new directives like noindex or nofollow, you need to ensure Screaming Frog is re-fetching everything fresh from the live site. This is crucial when validating on-page elements, checking googlebot accessibility, or verifying that search engine crawlers can properly access your content.
Preventing Cross-Project Contamination
When working on multiple projects – particularly when testing staging versus production environments with different user-agent settings or crawl depth configurations – cached data from one project can inadvertently influence another, leading to misleading metrics in your google search console (GSC) reports or google analytics data.
Where Screaming Frog Stores Cache Files
Understanding cache locations is essential for effective management. Screaming Frog stores data in two primary locations:
Default Application Cache:
- Windows:
C:\Users\[YourUsername]\.ScreamingFrogSEOSpider\cache - macOS:
/Users/[YourUsername]/.ScreamingFrogSEOSpider/cache - Linux:
/home/[YourUsername]/.ScreamingFrogSEOSpider/cache
Note the dot (.) before the folder name – this indicates a hidden directory. On Windows, enable “Show hidden items” in File Explorer. On Mac, press Cmd + Shift + . in Finder.
Project-Specific Cache:
Each saved .seospider project file is actually a folder containing the crawl’s database, downloaded resources, screenshots, and cached response codes. Screaming Frog stores this data in a ProjectInstanceData folder within each project directory. These project-specific caches are located wherever you save your projects and typically consume the most disk space. On systems using SSD disk storage, this is less of a performance concern, but on traditional hard drives, a bloated ProjectInstanceData folder can noticeably slow down your computer performance and the SEO Spider itself.
Three Methods to Clear Screaming Frog Cache
Method 1: In-App Cache Clearing (Recommended)
This is the safest approach for general maintenance and troubleshooting:
- Close all open projects: Go to File > Close Project
- Navigate to Configuration: Click Configuration in the top menu
- Select Clear Cache: Choose the Clear Cache option
- Confirm: Accept the confirmation dialog
- Restart Screaming Frog (optional but recommended)
This method clears the application-wide cache but preserves your saved project files. It’s ideal for resolving general performance issues without losing historical crawl data.
Method 2: Manual Cache Deletion
For more thorough cleaning or when the in-app method doesn’t resolve issues:
- Completely close Screaming Frog
- Navigate to the cache directory:
- Windows: Type %USERPROFILE%\.ScreamingFrogSEOSpider in File Explorer
- macOS: Press Cmd + Shift + G and type ~/.ScreamingFrogSEOSpider
- Linux: Navigate to ~/.ScreamingFrogSEOSpider in your file manager
- Open the cache folder
- Delete all contents (not the folder itself)
- Optional: Also clear the temp folder contents
- Empty your Recycle Bin/Trash
Important: Ensure Screaming Frog is fully closed before deleting files to prevent corruption. This method is more aggressive and can help resolve persistent issues that survive the in-app clearing.
Method 3: Deleting Project-Specific Cache
To reclaim significant disk space or remove a problematic project entirely:
- Close Screaming Frog completely
- Locate your saved project file (e.g., MyAudit.seospider)
- Inside the project directory, you’ll find the ProjectInstanceData folder containing the crawl database and all cached website scans. This is where the bulk of the disk space is consumed.
- Delete the entire project folder
- Empty your Recycle Bin/Trash
Critical Warning: This permanently deletes all crawl data for that project, including the number of urls crawled, anchor text analysis, pagination data, redirect chains, and all metrics. Only delete projects you’re certain you no longer need.
Method 4: Restart Your Computer (The Nuclear Option)
Sometimes the simplest solution is the most effective. A full system restart can resolve cache-related issues that other methods miss.
When to use this method:
- Stubborn file locks: Cache files show as “in use” and can’t be deleted even with Screaming Frog closed
- Memory not releasing: Task Manager (Windows) or Activity Monitor (Mac) shows Screaming Frog processes still consuming RAM after closing
- Post-manual deletion: After manually clearing cache folders, a restart ensures all system-level temp files are also purged
- Persistent crashes: When Screaming Frog crashes repeatedly despite clearing cache through other methods
- After major updates: Following Screaming Frog updates or Windows/Mac OS updates that might affect crawling functionality
Why it works:
Restarting clears system-level caches, releases all file locks, purges temporary memory allocations, resets network connections, and eliminates any zombie processes that might be interfering with Screaming Frog. It’s essentially giving both your operating system and Screaming Frog a completely fresh start.
Best practice: Combine this with Method 2 (Manual Cache Deletion). Clear the cache folders, restart your computer, then open Screaming Frog fresh. This combination resolves roughly 80% of persistent cache-related issues.
Pro tip: If you’re regularly experiencing issues that only restart fixes, it indicates underlying problems – insufficient RAM allocation, disk fragmentation on HDDs, or system instability that should be addressed separately.
Best Practices for Cache Management
Optimize Before You Crawl
Memory Allocation: Configure Configuration > Memory to allocate 60-80% of your available RAM. More RAM means less disk-based caching and faster crawls.
Storage Mode: For crawls exceeding 100,000 URLs, use Database Storage (Configuration > Storage) instead of Memory Storage. This writes data directly to the ProjectInstanceData folder on disk, preventing out-of-memory crashes while crawling sites with extensive internal links and complex content-type variations. Be aware that database storage mode will cause the ProjectInstanceData folder to grow significantly with each crawl, so monitor your available disk space accordingly.
Chrome Rendering: If you’re using javascript rendering features (which simulate Chrome browser behavior), be aware this significantly increases cache size due to storing rendered page data and screenshots.
Strategic Project Management
Archive, Don’t Hoard: Move completed projects to external storage or cloud backup. Keep only active projects and baseline crawls needed for comparison.
Regular Maintenance: Monthly cache clearing prevents buildup. Schedule this during low-activity periods.
List Mode Projects: For bulk export operations or quick list mode crawls, consider not saving these as project files unless you need the data long-term.
Troubleshooting Persistent Issues
If clearing the cache doesn’t resolve problems:
Check RAM Allocation: Verify Configuration > Memory settings. Insufficient memory allocation causes excessive disk caching and instability.
Update Java: Screaming Frog depends on Java. Ensure you’re running the recommended version for your operating system.
Corrupted Project File: If one specific project crashes repeatedly, the project database may be corrupted. Try opening it on another machine to confirm, or accept the loss and re-crawl.
Disk Space: Ensure your primary drive has at least 15-20% free space. Operating systems need room for swap files and temporary operations.
Firewall/Antivirus: Security software can block Screaming Frog’s network requests or interfere with file operations. Add exceptions if needed.
Plugin Conflicts: If you’re using third-party plugins or custom regex configurations for advanced validation, these can occasionally cause conflicts with cached data.
Advanced Tips
SSD vs. HDD: Install Screaming Frog and store projects on an SSD for dramatically faster performance. The constant read/write operations during crawls benefit enormously from SSD speeds.
User-Agent Settings: When testing how different crawlers (googlebot, bingbot, etc.) see your site, clear the cache between tests to ensure accurate results that reflect what search engines actually encounter.
API Usage: If you’re using Screaming Frog’s API for automated crawls, implement automated cache clearing in your workflow to prevent buildup from scheduled crawls.
XPath and Custom Extraction: When using xpath or regex for custom data extraction from HTML, cache issues can cause extraction functions to miss updated content. A fresh crawl ensures accurate dependency tracking.
Conclusion
Clearing Screaming Frog’s cache isn’t just routine maintenance – it’s essential for accurate SEO audits, reliable metrics, and optimal performance. Whether you’re analyzing duplicate content, validating meta tag implementations, tracking serp performance, or conducting comprehensive on-page analysis, a clean cache ensures your data reflects current site conditions.
Start with the in-app method for routine cleaning, use manual deletion for persistent issues, and only delete project-specific caches when you’re certain you don’t need the historical data. Implement proper memory allocation, use database storage for large crawls, and maintain regular cleaning schedules to keep Screaming Frog running at peak efficiency.
With these techniques, you’ll spend less time troubleshooting and more time extracting actionable SEO insights from your crawls.
Published on: 2026-01-02
Updated on: 2026-04-14
