Step 1: Separate “not indexed at all” from “indexed, but not ranking for your query”
Before changing a single line of code, determine whether Google knows your website exists:
- Test A (`site:` operator): Type
site:yourdomain.com(with no spaces after `site:` and no `https://`) into Google Search. While `site:` is not a complete census, seeing zero results versus seeing 40 pages immediately tells you whether the entire domain is excluded. - Test B (Google Search Console URL Inspection): Open Google Search Console, verify your domain as a **Domain property (`sc-domain:yourdomain.com`)**, and paste your homepage URL into the top inspection bar.
If Search Console says **URL is not on Google**, you have a discovery, crawling, or indexing block (Steps 2–6 below). If Search Console says **URL is on Google**, your technical plumbing is working, and your challenge is search relevance, internal linking, or content depth (Steps 7–10).
02 / THE 5 HARD TECHNICAL BLOCKERSSteps 2–6: The 5 technical mistakes that keep a website completely out of Google
When an entire site or newly launched section fails to appear in Google, one of these five configurations is almost always responsible:
| Step & Blocker | Where It Hides | Exact Fix |
|---|---|---|
Step 2: Leftover noindex Directive |
HTML <meta name="robots" content="noindex">, CMS “Discourage search engines” checkbox, or hosting X-Robots-Tag: noindex header. |
Remove the `noindex` directive so production pages serve `index, follow`, then request re-indexing in GSC. |
Step 3: robots.txt Blocking Crawlers |
https://yourdomain.com/robots.txt contains User-agent: * followed by Disallow: / from development. |
Update `robots.txt` to `Allow: /` for `Googlebot`, `Bingbot`, `OAI-SearchBot`, and `*`, and add your `Sitemap:` URL. |
| Step 4: Broken Canonical URL | <link rel="canonical"> points to `http://localhost:3000`, a `*.vercel.app` / `*.pages.dev` preview domain, or the homepage `/` on every child page. |
Configure `metadataBase` or your CMS canonical generator so every page outputs an absolute, self-referencing production canonical URL. |
| Step 5: Wrong GSC Property Variant | You verified `http://yourdomain.com` in Search Console, while your live site redirects to `https://www.yourdomain.com`. | Add a **Domain property (`sc-domain:yourdomain.com`)** via DNS TXT verification so Search Console tracks all protocols and subdomains together. |
| Step 6: Client-Side Empty HTML Shell | Single-Page Application (React/Vite/Vue) renders <div id="root"></div> with client-only data fetching that fails or times out during rendering. |
Pre-render or server-render (SSR/SSG) critical page titles, headings, body text, and internal <a href> links in the initial HTML response. |
Steps 7–10: Why your pages are indexed (or crawled) but don’t appear when people search
If your technical checks pass and your sitemap is submitted, check these four editorial and structural signals:
-
Step 07
Your title tag and H1 don’t match how people actually search on Google
If your page title is a vague brand slogan (*“The Future of Workflow Synergy”*) instead of the specific problem users type into Google (*“GitHub SEO Automation & Markdown Article Workflow”*), Google has no clear lexical or semantic signal to rank your page for that query.
-
Step 08
Pages are stuck in “Discovered” or “Crawled – currently not indexed”
Check **Indexing → Pages** in Search Console. If pages are *Discovered – currently not indexed*, they lack internal links from your homepage and category hubs. If they are *Crawled – currently not indexed*, Google fetched them and judged the content too thin or duplicative compared to existing pages.
-
Step 09
Brand-new domain timeline & lack of external citations
A newly registered domain with zero backlinks from other websites typically takes several days to a few weeks for Google to crawl and evaluate beyond the homepage. Submitting your XML sitemap in Search Console and Bing Webmaster Tools speeds up discovery, but cannot bypass initial quality evaluation.
-
Step 10
Manual actions or inherited domain penalties
If you bought an expired or previously owned domain, check **Security & Manual Actions → Manual actions** in Search Console. If the previous owner used the domain for spam, you will need to clean up the site and submit a reconsideration request.
How to check your live HTTP headers and HTML tags from the terminal
Browser extensions often inspect the hydrated DOM rather than the raw HTTP response Googlebot receives. Run these two commands in your terminal (or run an automated audit in Searchbop) to see the exact truth:
# 1. Check HTTP status code, redirect chain, and X-Robots-Tag header
curl -ILs https://yourdomain.com/ | grep -iE "^HTTP/|location:|x-robots-tag:"
# 2. Check rendered HTML <title>, meta robots, and canonical link tag
curl -s https://yourdomain.com/ | grep -iE "<title>|name=\"robots\"|rel=\"canonical\""
Official Google Search Central documentation
Reference documentation: See Google’s official guide Why is my page missing from Google Search? and the Crawling and Indexing FAQ. Remember that crawling and indexing can take anywhere from a few days to several weeks after publishing or fixing a technical blocker, and no tool can guarantee indexing or ranking.
Research context & search queries covered on this page Indexing · P1
Cluster & intent: indexing & audit (Informational · P1) · Target queries: why is my website not showing on Google, Google indexing troubleshooting, website indexing checker, sitemap submitted but not indexed.
Related benchmark prompts: P01, P11, and P12.