Scrape Google SERPs with Screaming Frog

Scrape Google SERPs with Screaming Frog

Before you start, you might have a few questions:

Is it legal to scrape Google?

Google does not take legal action against scraping, likely for self-protective reasons. Google does test the User-Agent (The browser type) and automatically rejects User-Agents that seem to originate from possible automated bots.

Why do we want to scrape Google search results?

Analyzing Search Engine Results Pages (SERPs) provides insight into the top search results for a specific keyword or topic. You can use this information to determine how to rank your website in search results better.

All that said don’t be an idiot and get your IP blocked by Google. Please, respect Google’s crawl limits (noted in the tutorial)

This tutorial will teach you how to scrape a Google SERP using Screaming Frog.

Configure Screaming Frog

Set Screaming Frog to List Mode

List Mode
List Mode

Uncheck all items in Configuration > Spider > Crawl

Spider Configuration
Spider Configuration

Add XPath Selectors in Screaming Frog

Set your Custom Extraction using the settings below.

Custom Extraction
Custom Extraction

Then add the XPath Name, Selector, Code, + “Extract Text”

Extraction Settings
Extraction Settings
#Page Titles
//h3[@class="LC20lb DKV0Md"]
 
#URLs
//div[@class="r"]/a[1]/@href
 
#Meta Descriptions
//div[@class="s"]/div/span[@class="st"]

Set your User-Agent

Under Configuration > User-Agent, set a user-friendly agent.

User-Agent Settings
User-Agent Settings

Chrome, Firefox, Safari shouldn’t give you any issues.

User-Agent Configuration
User-Agent Configuration

Respect Google

Don’t crawl too fast. In Configuration > Speed, set the Max Threads to 1.0, checked the Limit URL/s, and set the Max URL/s to 0.8.

Thread Speed
Thread Speed

Enter your Crawl

The last step, now you need to enter your search query. Under Upload > Enter Manually

Crawl Settings
Crawl Settings
#Google Query
https://www.google.com/search?q=your+keyword+phrase
 
#Number
https://www.google.com/search?q=your+keyword+phrase&num=50
 
#Location
https://www.google.com/search?q=your+keyword+phrase&num=50&near=chicago,+il

The results will show up horizontally

Crawl Results
Crawl Results

To pull “People also ask”, I recommend using Ahrefs or AlsoAsked.com to get accurate results.

Transpose feature in Excel

Since the organic results are horizontal, you can Export the results into a CSV. In Excel, you will want to use the Transpose feature. In Excel, select the desired cells you would like to change and copy (CMD+C on a Mac). Then choose Paste Options > Transpose.

Transpose in Excel
Transpose in Excel

Settings saved in Pastebin

You can learn more about web scraping in our custom extraction article.


Published on: 2021-03-10
Updated on: 2021-03-29

Avatar for Isaac Adams-Hands

Isaac Adams-Hands

Isaac Adams-Hands is an SEO Director, Full Stack Developer, and InfoSec enthusiast. He received his Bachelor’s Degree from the University of Western Sydney before joining various marketing positions in search portals, higher education, and addiction recovery marketing agencies.