On the 31st of January, Google released their Search Console URL Inspection API to the public, announcing the news in a post on the Google developers’ forum.

The new Search Console URL Inspection API contains a number of fields and parameters that will undoubtedly assist the technical reporting of site health. In order to aid the work of SEOs, specifically in the context of clear, easy, and understandable reporting of this data to their stakeholders, I created a Data Studio Dashboard for reporting the URL Inspection data.

In the post below, we will discuss the main use-cases of the API, the different methods available for pulling the data, as well as how to set up (easily) the dashboard, and interpret the visualizations for enhanced technical performance reporting.

Overview of the URL Inspection API

The URL Inspection API uses an index.inspect method, which means that it enables you to view the indexed or indexable status of a URL you provided.

You can make a request with two parameters needed – the URL you’d like to inspect and the URL of the property as defined in Search Console.

The request body should contain data with the following structure:

JSON representation
{ “inspectionUrl”: string, “siteUrl”: string, “languageCode”: string }

The response will also be returned in JSON format, which makes it super easy for manipulation and visualization afterward.

Exclusions:

• Presently only the status of the version in the Google index is available; you cannot test the indexability of a live URL.

Quota limits:

The quota is enforced per Search Console website property (calls querying the same site):

• 2,000 queries per day
• 600 queries per minute

Check out the Search Console APIs documentation usage limits for more detailed guidance.

Also, be sure to check out the list of parameters returned by the Google Search Console URL Inspection API.

Before we dive in, a quick word on the different methods that are available for extracting data from the URl inspection API.

Different methods for pulling the data from the URL Inspection API

If you have already read Aleyda‘s article on the 8 different tools you can use with this API, then feel free to skip this section. Read a summary of the free tools for data extraction below, but definitely check Aleyda’s article for a detailed review of all of these tools (and more!):

• Google Bulk Inspect URLs by Valentin Pletzer – free online tool, which enables a download of the data
• MyDomain.Dev by Lino Uruñuela – a free online tool, which enables a download of the data
• URL Inspection API in Sheets by Mike Richardson – Google Sheets Template using App Scripts, which can be connected directly in Data Studio
• Screaming Frog SEO Spider – a crawler, which enables an export of the data in sheets format
• Sitebulb – another SEO crawler, which enables data exports of the URL inspection data in sheets
• Google Index Inspection API Node.js script by Jose Luis Hernando – a JS script for pulling the data from the API
• Google URL Inspection API with Python by Jean-Christoph Chouinard  – a Python script for pulling data from the API

Google Search Console URL Inspection API Explorer Dashboard in Google Data Studio

How to set-up Data Studio Reporting for the URL inspection API (free dashboard)

Crawling your website

For my dashboard, I’ve used an export from my beloved tool Screaming Frog. In order to replicate my data source, feel free to run a crawl of your site, enabling the URL inspection API in the Configuration > API > Google Search Console API tab (after you’ve authenticated with the account you want to crawl).

Then, just run the crawl, go to the Search Console tab, and click export. Export the file as a Google Sheet in your connected Google Drive Account.

Then, access the Data Studio Dashboard from the link below and make a copy, connecting your exported crawl data as the data source.

? Notes on setting up

Sometime Google Data Studio malfunctions when copied, specifically with custom fields, filters, or filter application – I’ve previously talked about the limitations of Google Data Studio, however, to spare you, here is a summary of the custom fields, filters, and where they are applied, which can help you set up the dashboard:

• Custom Field: Canonical Mismatch? (Y/N)

• Custom Field:Page Section

• Custom Field:Pages with / without clicks

• Custom Filed: Crawl Frequency

• Filters added in the report:
  • Exclude pages with no mobile usability issues
  • Exclude pages with 0 clicks
  • Exclude URLs with no summary
  • Exclude Inspection Status is Null
  • Crawling not allowed Pages
  • Indexing Not allowed
  • Page Fetch Unsuccessful
  • Status Code = 404

Data Interpretation & Storytelling

Google Disclaimers on Interpreting Data

Before I begin with the section on how to interpret the data you see, here are a few important notes on data interpretation from Google:

• This is not a live test – as previously mentioned, the tool is reflective of most recent crawl, not the live page that exists. However, the information shown in the tool is used by Google to evaluate Search results.
• “URL is on Google” doesn’t actually mean that your page is appearing in Search results. Actual appearance in Search results requires that the page and its structured data conform to quality and security guidelines. The URL Inspection tool doesn’t take into account manual actions, content removals, or temporarily blocked URLs.
• URL not showing on Google but the tool says it is indexed? Explore this resource to find out why.
• The “Last crawl” date in the Coverage section shows the date when the information shown here was retrieved (if the page was successfully crawled).

Section Breakdown

Now, on with the fun stuff – using the dashboard! ? Below, I list some cool use-cases, as well as go through what each of the pages contains.

In the Coverage Overview Section, you can get an overview of:

• how many pages are crawled, as well as the distribution of pages per section
• how many pages have coverage status as per the Inspection API, including the distribution of pages per status
• how many pages receive clicks
• how many pages have mobile usability issues as well as the issues breakdown distribution
• how many pages have rich result issues as well as the error breakdown distribution
• a summary of index status distribution (options being: URL is on Google, URL is on Google but has issues, and URL is not on Google)
• a summary of crawl user agent distribution – is your site crawled mostly by mobile, desktop or unknown agent?
• a summary of page fetch status distribution – Have pages, included in the crawl been fetched successfully by Google?
• a summary of URL Inspection status distribution – have there been any errors?

In the Mobile Usability Section, you can:

• Get an Overview of whether your pages are mobile friendly or not (distribution)
• Get an overview of the main types of errors encountered on the site, and how many pages trigger them
• Filter URLs based on Error type, Indexability, and Google Index Status in order to help with prioritisation of fixes

In the Rich Result Snippet Issues section, you can:

• get an overview of the main errors, realted to rich snippets
• get an overview of the number of pages that are valid for rich results
• see the main types of schema types implemented
• filter and explore URLs, based on indexability status, clicks, and Google Index Status to help with prioritising fixes.

Now, moving on to Page 2 of the dashboard, which addresses Indexability, Canonicalisation, and Crawl Behaviour.

In the Google Status Section you can:

• Filter and explore URLs, based on URL inspection API status, Summary, Pages with/ Without Clicks, and Coverage to help you get an idea on the type of pages on Google
• get an overview of the relationship between indexability & status code – are all of the pages you declared indexable returning a 200 status code?
• get an overview of the relationship between Google Index status and clicks – are pages with issues receiving clicks? Prioritise their optimisations.
• get an overview of the relationship between inspection status and errors breakdown – when an inspection error is received, what is the reason for this? Read timeout, Connect time? Investigate.

In the Crawling section, you can:

• get summary stats on
  • # of pages where crawling is not allowed
  • # of pages where indexing is not allowed
  • # of pages where unsuccessful page fetch is not allowed
  • # of 404s

• filter URLs based on crawl agent (mobile, desktop, unknown) – try to see if you find any patterns.
• filter based on last drawl date using the date range filter
• filter pased on page fetch status
• filter, based on status code
• get an overview of the crawling behaviour on the site – how frequently are URLs crawled?

In the Canonical Tags section, you can check for mismatches between the canonical tags you’ve indicated and the ones Google has selected. You can also integrate this with additional filters above the table to find patterns in any mistakes you might have made in declaring canonicals.

Executive Q&A

In-part inspired by Aleyda’s thread on questions the URL Inspection API answers, I wanted to introduce this final section on how to get the replies from the dashboard, which can enable you to better handle such questions on client calls.

• For which pages Google is not using the specified canonical configuration and what is using instead?

Go to page two, filter the table in the Canonical Tags section, using the filter Canonical Mismatch (Y/N) by setting it to yes. You can now quickly review any mismatches.

• Which ranked pages attracting clicks haven’t been crawled lately?

Go to page 2.

Select from the section Google Status, the filter Pages with/without clicks by setting it to show only pages with clicks.

Then scroll down to the next section, and use the date bar, in order to adjust the time-frame you want to review. Then, you can see the result in the table below with URLs, Last Crawl (Date), and number of clicks.

Check if these URLs have been submitted in the XML sitemap.

• When was the last time Google crawled your URLs? How?

You can answer all of these questions with the Crawling section on page 2.

• Which indexable URLs attracting clicks are triggering mobile usability or rich results issues?

Go to page 1, in sections Mobile Usability Issues and Rich Result Snippet Issues, and use the Indexability filters to answer these questions.

Summary

The Google URL inspection API was released soon, and the industry has shown it lots of love so far.

While there are some limitations related to quota, usage, and exclusions, the API is a much-needed extension to the dataset of SEOs.

There are at least 8 different ways to export data, many of which will enable you to visualise it using the Google Search Console URL Inspection API Explorer Data Studio Dashboard I have built.

The Data Studio Dashboard consists of several different sections: Coverage Overview, Mobile Usability Issues, Rich Result Snippet Issues, Google Status, Crawling, Canonical Tags, which will help you answer a ton of questions about the sites you are reporting on.

Feel free to let me know how I can improve it, or let me know how you’d use it. Thanks ?

Frequently Asked Questions