01 / WHY SITE: SCRAPERS FAIL

Why `site:yourdomain.com` is not a reliable website indexing checker

Google Search Central has repeatedly documented that the `site:` search query operator is not designed for exhaustive index auditing:

02 / THE URL INSPECTION API PAYLOAD

The 7 fields that matter in the Google Search Console URL Inspection API

When you connect your Google Search Console property in Searchbop, the desktop worker calls `https://searchconsole.googleapis.com/v1/urlInspection/index:inspect` for your sitemap and audited URLs and extracts seven critical fields from `inspectionResult.indexStatusResult`:

API Field Example Value What It Tells You
verdict PASS | NEUTRAL | FAIL Whether the URL is indexed and eligible to appear in Google Search (`PASS`), excluded (`NEUTRAL`), or blocked by an error (`FAIL`).
coverageState "Submitted and indexed" / "Crawled - currently not indexed" The exact human-readable coverage bucket shown in the Search Console UI.
indexingState INDEXING_ALLOWED | BLOCKED_BY_META_TAG | BLOCKED_BY_HTTP_HEADER Immediately reveals whether a <meta name="robots" content="noindex"> or `X-Robots-Tag` blocked indexing.
robotsTxtState ALLOWED | DISALLOWED Whether your `robots.txt` allowed Googlebot to fetch the URL at crawl time.
pageFetchState SUCCESSFUL | SOFT_404 | NOT_FOUND | SERVER_ERROR Whether your server returned a healthy `200 OK` page or failed during Googlebot’s fetch.
lastCrawlTime "2026-10-30T08:19:44Z" (or omitted) Proves whether Googlebot has fetched the latest version of your page since your last deployment.
userCanonical vs. googleCanonical https://example.com/a/ vs. https://example.com/b/ Reveals when Google ignored your <link rel="canonical"> tag and folded your page into a different URL.
03 / CATCHING CANONICAL OVERRIDES

Spotting when `googleCanonical` overrides your `userCanonical`

One of the highest-value checks in Searchbop’s indexing audit is comparing `userCanonical` (the <link rel="canonical"> tag you declared in your HTML) against `googleCanonical` (the URL Google actually chose to index). If those two URLs differ, your page is excluded under *Duplicate, Google chose different canonical than user*—usually because internal links, sitemap entries, or near-identical page copy convinced Google that another URL is the primary version.

URL INSPECTION API JSON RESPONSE (SIMPLIFIED)How Searchbop detects a canonical override and stale crawl
{
  "inspectionResult": {
    "indexStatusResult": {
      "verdict": "NEUTRAL",
      "coverageState": "Duplicate, Google chose different canonical than user",
      "robotsTxtState": "ALLOWED",
      "indexingState": "INDEXING_ALLOWED",
      "lastCrawlTime": "2026-10-25T16:04:12Z",
      "pageFetchState": "SUCCESSFUL",
      "googleCanonical": "https://tengrx.com/guides/nextjs-seo-audit/",
      "userCanonical": "https://tengrx.com/guides/nextjs-canonical-url/",
      "sitemap": ["https://tengrx.com/sitemap.xml"]
    }
  }
}
04 / QUOTA MANAGEMENT & LOCAL CACHING

How Searchbop respects Google’s URL Inspection API quotas automatically

Google enforces a strict quota on the URL Inspection API (2,000 inspection requests per day and 600 per minute per Search Console property). Instead of re-querying every URL on every scan and exhausting your quota, Searchbop stores each URL’s inspection verdict and `checkedAt` timestamp in your local SQLite database. During an audit run, it prioritizes uninspected URLs and URLs whose cached verdict is older than 24 hours (inspecting up to 100 URLs per run with controlled concurrency).

Official API documentation & boundaries

What the URL Inspection API can and cannot do: The official Google Search Console URL Inspection API is read-only: it reports Google’s current indexed state for a URL, but does not offer an API endpoint to trigger “Request Indexing” in bulk (for regular web pages, Google uses XML sitemaps and natural crawling for discovery). Reference: Search Console API Usage Limits.

Research context & search queries covered on this page Audit · P0

Cluster & intent: audit & indexing (Commercial / Informational · P0/P1) · Target queries: website indexing checker, Google Search Console analysis tool, AI SEO audit, SEO audit tool, technical SEO audit software.

Related benchmark prompts: P01, P11, and P12.