You may have heard of Screaming Frog SEO Spider is the go-to tool for analyzing and finding links to your website. It’s a powerful web crawler that allows you to export data from websites, find broken links, check title tags, meta descriptions, and more. But did you know that it can also be used for Google SERP scraping? In this article, we will show how to scrape Google SERPs with Screaming Frog so you can get all the information about what people are searching for in one place!
Before you start, you might have a few questions:
Is it legal to scrape Google?
Google does not take legal action against scraping, likely for self-protective reasons. Google does test the User-Agent (The browser type) and automatically rejects User-Agents that seem to originate from possible automated bots.
Why do we want to scrape Google search results?
Analyzing Search Engine Results Pages (SERPs) provides insight into the top search results for a specific keyword or topic. You can use this information to determine how to rank your website in search results better.
All that said don’t be an idiot and get your IP blocked by Google. Please, respect Google’s crawl limits (noted in the tutorial)
This tutorial will teach you how to scrape a Google SERP using Screaming Frog.
Configure Screaming Frog
Set Screaming Frog to List Mode
Uncheck all items in Configuration > Spider > Crawl
Add XPath Selectors in Screaming Frog
Set your Custom Extraction using the settings below.
Then add the XPath Name, Selector, Code, + “Extract Text”
#Page Titles
//h3[@class="LC20lb DKV0Md"]
#URLs
//div[@class="r"]/a[1]/@href
#Meta Descriptions
//div[@class="s"]/div/span[@class="st"]Set your User-Agent
Under Configuration > User-Agent, set a user-friendly agent.
Chrome, Firefox, Safari shouldn’t give you any issues.
Respect Google
Don’t crawl too fast. In Configuration > Speed, set the Max Threads to 1.0, checked the Limit URL/s, and set the Max URL/s to 0.8.
Enter your Crawl
The last step, now you need to enter your search query. Under Upload > Enter Manually
#Google Query
https://www.google.com/search?q=your+keyword+phrase
#Number
https://www.google.com/search?q=your+keyword+phrase&num=50
#Location
https://www.google.com/search?q=your+keyword+phrase&num=50&near=chicago,+ilThe results will show up horizontally
To pull “People also ask”, I recommend using Ahrefs or AlsoAsked.com to get accurate results.
Transpose feature in Excel
Since the organic results are horizontal, you can Export the results into a CSV. In Excel, you will want to use the Transpose feature. In Excel, select the desired cells you would like to change and copy (CMD+C on a Mac). Then choose Paste Options > Transpose.
All search commands in one spot:
Now that you’ve learned how to scrape Google SERPs with Screaming Frog, here are a few other uses for screaming frog:
- How To Analyze Link Position with Screaming Frog
- Screaming Frog’s Custom Extractions
- What is Screaming Frog?
Published on: 2021-03-10
Updated on: 2021-06-09
