You may have heard of Screaming Frog SEO Spider is the go-to tool for analyzing and finding links to your website. It’s a powerful web crawler allows you to export data from websites, find broken links, check title tags, meta descriptions, and more. But did you know that it can also be used for Google SERP scraping without an API? This article will show beginners how to scrape Google SERPs with Screaming Frog so you can get all the information about what people are searching for in one place!
Before you start, you might have a few questions:
Table of Contents
- What is SERP mode in Screaming Frog?
- SERP Snippet Emulator and Editing
- Is it legal to scrape Google?
- Why do we want to scrape Google search results?
- Alternative Scraping Tools
- Configure Screaming Frog
- Add XPath Selectors in Screaming Frog
- Transpose feature in Excel
- API Integrations for Enhanced Data
- All search commands in one spot:
- Did this article answer your questions?
What is SERP mode in Screaming Frog?
SERP mode allows you to export page titles and descriptions from Screaming Frog, make bulk edits in Excel and then upload them back into the tool to understand how they may appear in Google’s SERPs.
SERP Snippet Emulator and Editing
The scraping method covered in this tutorial uses List Mode with custom XPath extraction to pull live SERP data from Google. Screaming Frog also has a dedicated SERP mode that serves a different purpose: previewing and editing how your own page titles and meta descriptions will appear in search results.
In SERP mode, you upload a CSV or Excel file containing your URLs, titles, and meta descriptions. Screaming Frog then calculates pixel widths and character lengths for each snippet and shows you a visual preview of how they’ll render on desktop, mobile, and tablet. No crawling is involved, so it’s fast.
This is useful when you’re making bulk metadata changes across a site. You can edit titles and descriptions directly in the tool, see if they’ll get truncated on any device, and export the updated metadata to hand off for implementation.
Think of it this way: List Mode scraping (the method in this tutorial) is for pulling data from Google. SERP mode is for refining your own metadata before it goes live.
Is it legal to scrape Google?
Google does not take legal action against scraping, likely for self-protective reasons. Google tests the User-Agent (The browser type) and automatically rejects User-Agents that seem to originate from possible automated bots.
Why do we want to scrape Google search results?
Analyzing Search Engine Results Pages (SERPs) provides insight into the top search results for a specific keyword or topic. You can use this information to rank your website in search results better.
All that said, don’t be an idiot and get your IP blocked by Google. Please, respect Google’s crawl limits (noted in the tutorial)
This tutorial will teach you how to scrape a Google SERP using Screaming Frog.
Alternative Scraping Tools
You can use open-source python alternatives like Beautifulsoup or Octoparse, but I find this method 10x easier to set up to scrape data.
Configure Screaming Frog
Set Screaming Frog to List Mode
Uncheck all items in Configuration > Spider > Crawl
Add XPath Selectors in Screaming Frog
Set your Custom Extraction using the settings below.
Then add the XPath Name, Selector, Code, + “Extract Text” from the webpage. This step will parse out the HTML and provide you with clean data. You can change these settings by looking at the source code to extract different items.
#Page Titles
//h3[@class="LC20lb DKV0Md"]
#URLs
//div[@class="r"]/a[1]/@href
#Meta Descriptions
//div[@class="s"]/div/span[@class="st"]Set your User-Agent
Under Configuration > User-Agent, set a user-friendly agent.
Chrome, Firefox, Safari shouldn’t give you any issues.
Respect Google
Don’t crawl too fast. In Configuration > Speed, set the Max Threads to 1.0, check the Limit URL/s, and set the Max URL/s to 0.8. If you want to crawl faster, I recommend using proxies to hide your IP address and a captcha breaker, so Google doesn’t ban you.
Enter your Crawl
In the last step, now you need to enter your search query. Under Upload > Enter Manually.
This will also work on scraping LinkedIn, Amazon, Bing with a little bit of tweaking.
#Google Query
https://www.google.com/search?q=your+keyword+phrase
#Number
https://www.google.com/search?q=your+keyword+phrase&num=50
#Location
https://www.google.com/search?q=your+keyword+phrase&num=50&near=chicago,+ilThe results will show up horizontally
To pull “People also ask,” I recommend using Ahrefs or AlsoAsked.com to get accurate results.
Transpose feature in Excel
In Excel, you will want to use the Transpose feature to filter your web scraping results. Select the desired cells you would like to change and copy (CMD+C on a Mac) to transpose the data. Then choose Paste Options > Transpose. Since the organic results are horizontal, you can export the results into a CSV.
The same method works in Google Sheets to transpose Google Search Results.
API Integrations for Enhanced Data
Screaming Frog connects to several third-party APIs that can enrich your crawl data beyond what you get from scraping SERPs alone. Under Configuration > API Access, you can integrate with Google Search Console, Google Analytics 4, PageSpeed Insights, Ahrefs, Semrush, and Moz Pro.
For SERP scraping workflows, the most useful integration is Google Search Console. Once connected, you can pull in click data, impressions, and average position alongside the titles and descriptions you’ve extracted. This lets you compare what’s actually ranking against what you’re seeing in the live SERPs.
Google Analytics 4 integration adds session and engagement data, so you can identify pages where SERP appearance might be hurting click-through rates despite decent rankings. PageSpeed Insights pulls in Core Web Vitals scores, which is helpful if you’re auditing competitor pages from your scraped results.
You don’t need any of these APIs to follow the scraping tutorial above, but connecting them turns a one-dimensional SERP export into a more complete dataset for analysis.
All search commands in one spot:
Now that you’ve learned how to scrape Google SERPs with Screaming Frog, here are a few other uses for screaming frog:
- How To Analyze Link Position with Screaming Frog
- Screaming Frog’s Custom Extractions
- What is Screaming Frog?
FAQs
How to scrape Google?
How to scrape search results?
Published on: 2021-03-10
Updated on: 2026-04-02
