How to scrape search results?

There are a few ways to scrape search results. One way is to use a web scraper, which is a piece of software that extracts data from websites. Another way is to use a public API, which is a set of programming instructions that allow you to access data from a website. Finally, you can manually copy and paste the data from the search results page into a spreadsheet or text editor. Whichever method you choose, make sure to follow the instructions carefully so that you don't accidentally block yourself from accessing the website's data.

Google Scraper - Scrape Google SERPs with Screaming Frog

Q: Why do we want to scrape Google search results?

Analyzing Search Engine Results Pages ( SERPs ) provides insight into the top search results for a specific keyword or topic. You can use this information to rank your website in search results better. Read More

Q: Configure Screaming Frog

Set Screaming Frog to List Mode Read More

Q: Add XPath Selectors in Screaming Frog

Set your Custom Extraction using the settings below. Read More

Scrape Google SERPs with Screaming Frog — Google Scraping

You may have heard of Screaming Frog SEO Spider is the go-to tool for analyzing and finding links to your website. It’s a powerful web crawler allows you to export data from websites, find broken links, check title tags, meta descriptions, and more. But did you know that it can also be used for Google SERP scraping without an API? This article will show beginners how to scrape Google SERPs with Screaming Frog so you can get all the information about what people are searching for in one place!

Before you start, you might have a few questions:

Table of Contents

What is SERP mode in Screaming Frog?
Is it legal to scrape Google?
Why do we want to scrape Google search results?
Alternative Scraping Tools
Configure Screaming Frog
Add XPath Selectors in Screaming Frog
Transpose feature in Excel
All search commands in one spot:
FAQ
Did this article answer your questions?

What is SERP mode in Screaming Frog?

SERP mode allows you to export page titles and descriptions from Screaming Frog, make bulk edits in Excel and then upload them back into the tool to understand how they may appear in Google’s SERPs.

Is it legal to scrape Google?

Google does not take legal action against scraping, likely for self-protective reasons. Google tests the User-Agent (The browser type) and automatically rejects User-Agents that seem to originate from possible automated bots.

Why do we want to scrape Google search results?

Analyzing Search Engine Results Pages (SERPs) provides insight into the top search results for a specific keyword or topic. You can use this information to rank your website in search results better.

All that said, don’t be an idiot and get your IP blocked by Google. Please, respect Google’s crawl limits (noted in the tutorial)

This tutorial will teach you how to scrape a Google SERP using Screaming Frog.

Alternative Scraping Tools

You can use open-source python alternatives like Beautifulsoup or Octoparse, but I find this method 10x easier to set up to scrape data.

Configure Screaming Frog

Set Screaming Frog to List Mode

Uncheck all items in Configuration > Spider > Crawl

Add XPath Selectors in Screaming Frog

Set your Custom Extraction using the settings below.

Then add the XPath Name, Selector, Code, + “Extract Text” from the webpage. This step will parse out the HTML and provide you with clean data. You can change these settings by looking at the source code to extract different items.

#Page Titles
//h3[@class="LC20lb DKV0Md"]
 
#URLs
//div[@class="r"]/a[1]/@href
 
#Meta Descriptions
//div[@class="s"]/div/span[@class="st"]

Set your User-Agent

Under Configuration > User-Agent, set a user-friendly agent.

Chrome, Firefox, Safari shouldn’t give you any issues.

Respect Google

Don’t crawl too fast. In Configuration > Speed, set the Max Threads to 1.0, check the Limit URL/s, and set the Max URL/s to 0.8. If you want to crawl faster, I recommend using proxies to hide your IP address and a captcha breaker, so Google doesn’t ban you.

Enter your Crawl

In the last step, now you need to enter your search query. Under Upload > Enter Manually.

This will also work on scraping LinkedIn, Amazon, Bing with a little bit of tweaking.

#Google Query
https://www.google.com/search?q=your+keyword+phrase
 
#Number
https://www.google.com/search?q=your+keyword+phrase&num=50
 
#Location
https://www.google.com/search?q=your+keyword+phrase&num=50&near=chicago,+il

The results will show up horizontally

To pull “People also ask,” I recommend using Ahrefs or AlsoAsked.com to get accurate results.

Transpose feature in Excel

In Excel, you will want to use the Transpose feature to filter your web scraping results. Select the desired cells you would like to change and copy (CMD+C on a Mac) to transpose the data. Then choose Paste Options > Transpose. Since the organic results are horizontal, you can export the results into a CSV.

The same method works in Google Sheets to transpose Google Search Results.

All search commands in one spot:

Now that you’ve learned how to scrape Google SERPs with Screaming Frog, here are a few other uses for screaming frog:

How To Analyze Link Position with Screaming Frog
Screaming Frog’s Custom Extractions
What is Screaming Frog?

FAQ

How to scrape Google?
Google scraping is a process of extracting data from Google search results. It can be used to collect data for competitor analysis, keyword research, market research, and more. There are a few different ways to scrape Google, but the most common is to use a web scraper. A web scraper is a piece of software that simulates a user browsing the web. It sends requests to Google and then parses the response HTML to extract the data you need. Scraping Google requires some technical expertise, but it’s not difficult to learn. There are many tutorials and resources available online. Once you have a basic understanding of how it works, you can start scraping Google for your own purposes.
How to scrape search results?
There are a few ways to scrape search results. One way is to use a web scraper, which is a piece of software that extracts data from websites. Another way is to use a public API, which is a set of programming instructions that allow you to access data from a website. Finally, you can manually copy and paste the data from the search results page into a spreadsheet or text editor. Whichever method you choose, make sure to follow the instructions carefully so that you don’t accidentally block yourself from accessing the website’s data.

Published on: 2021-03-10
Updated on: 2024-09-16

Isaac Adams-Hands

Isaac Adams-Hands is the SEO Director at SEO North, a company that provides Search Engine Optimization services. As an SEO Professional, Isaac has considerable expertise in On-page SEO, Off-page SEO, and Technical SEO, which gives him a leg up against the competition.