How To Automate Screaming Frog Crawl Reports In Data Studio

Schedule a Crawl

With Screaming Frog open, go to File >> Scheduling.

Name & Set a Date

Under General Settings of Create Task, enter a Task Name, Project Name, and a Date/Time you’d like the crawl to run.

The Date/Time is when the crawl will run. Task Name will be used for the file name, and Project Name will be used for the Folder in G-Drive.

Crawl Seed

Enter the domain name you want crawled in the Crawl Seed settings under the Start Options tab.

Export Crawl

Select the headless option under the Export Tab.

Pick a folder you’d like the file saved under

If you haven’t already, add your Google Drive Account. When you click on the Manage button, a modal window will pop up and ask you to sign into your Google Account. After which it will ask you to allow Screaming Frog to make changes.

Allow all access and press continue.

Selected Crawl Options

The last setting is Export for Data Studio (Custom Crawl Overview) >> Configure.

To keep this tutorial simple, select all (>>) settings.

Crawl History

After clicking Ok on the Create Task. Next, you will want to ensure your crawl has run before proceeding. You can find this under History. It’s important you see an End date and Duration, if that part is empty, the scan has not started or maybe it hasn’t completed yet.

Check G-Drive File

Once the crawl has run, check your G-Drive for the file. It will be saved under a Screaming Frog Folder. Then check the G-Sheet to ensure the data has been populated.

Connecting to Google Data Studio

You will need to make a copy of the Screaming Frog’s Google Data Studio template

The Copy function is located in the top right corner of the template.

Create Data Source

From the dropdown, select “Create Data Source”

Google Sheets

Select the Google Sheets Option.

Connector Settings

Select your Screaming Frog File saved in Google Drive as seen in the screen grab above.

Date

Every Type should be set as a Number, EXCEPT Date, the Date row needs to set as Date.

Chrome UX report

Now that you have the first source set, leave the Chrome UX Report as is, Screaming Frog will prompt you after you click Copy Report.

Authorize Chrome UX Report

As the template copies, a modal window will pop up. Click Authorize Chrome UX Report.

View Crawl Overview Report

Now the Screaming Frog template has copied over, you can view your scan results in data studio.