01 / LOCKING DOWN *.PAGES.DEV

How to stop `*.pages.dev` preview and branch URLs from being indexed by Google

Every Cloudflare Pages project automatically receives a `your-project.pages.dev` subdomain in addition to your custom production domain (e.g., `yourdomain.com`). While Cloudflare Pages automatically attaches `X-Robots-Tag: noindex` to hash-prefixed preview deployments (`a1b2c3d4.your-project.pages.dev`), the root `your-project.pages.dev` production alias can still be crawled and indexed if an external link or leaked sitemap points to it.

To guarantee that search engines never index any `*.pages.dev` URL while leaving your custom production domain `index, follow`, place a `_headers` file in your static build output directory (`public/_headers` in Next.js/Astro, or `dist/_headers` in custom static builds):

CLOUDFLARE PAGES _HEADERS FILE (public/_headers)Blocking *.pages.dev indexing while caching static assets
# 1. Prevent Google & Bing from indexing the default *.pages.dev host or branch aliases
https://:project.pages.dev/*
  X-Robots-Tag: noindex, nofollow

https://*.your-project.pages.dev/*
  X-Robots-Tag: noindex, nofollow

# 2. Security & caching headers for production custom domain
/*
  X-Content-Type-Options: nosniff
  Referrer-Policy: strict-origin-when-cross-origin

/fonts/*
  Cache-Control: public, max-age=31536000, immutable
02 / REDIRECTS & CANONICAL NORMALIZATION

Configuring `_redirects` on Cloudflare Pages without redirect loops

Cloudflare Pages processes a plain-text `_redirects` file in your build output directory (supporting up to 2,000 static rules and 100 dynamic rules). Remember two critical rules from Cloudflare’s routing documentation:

  1. Redirects execute before `_headers`: If a path matches a rule in `_redirects`, Cloudflare returns the redirect response immediately without applying `_headers` rules for that request.
  2. Cloudflare Pages Functions bypass `_redirects`: If your project uses Pages Functions (`functions/`) or full-stack SSR middleware, paths handled by a Function do not automatically run through static `_redirects` rules—you must return the `301` redirect inside your Function or middleware.
CLOUDFLARE PAGES _REDIRECTS FILE (public/_redirects)Redirecting legacy blog slugs and *.pages.dev traffic to your canonical domain
# Redirect default pages.dev production host to canonical custom domain
https://your-project.pages.dev/* https://yourdomain.com/:splat 301

# Redirect consolidated or renamed SEO guides with 301 Permanent Redirects
/blog/old-aeo-guide /guides/llms-txt-vs-robots-txt/ 301
/features/audit /features/seo-audit/ 301
03 / MARKDOWN & MDX FRONTMATTER ARCHITECTURE

Markdown blog SEO: Designing frontmatter fields that map cleanly to HTML <head> and JSON-LD

Whether your Markdown blog uses Contentlayer, Next.js next-mdx-remote, Astro Content Collections, or a custom build script, search engines never see your .md or .mdx source file—they only see the rendered HTML <head>, semantic headings (<h1>–<h3>), and JSON-LD structured data produced by your template.

Every high-performing Markdown blog template should enforce six core frontmatter fields at build time:

Frontmatter Key Rendered HTML / SEO Output Validation Rule
title HTML <title>, og:title, <h1>, and JSON-LD headline 45–65 characters; lead with the primary search query people type into Google.
description <meta name="description">, og:description, and JSON-LD description 130–160 characters; summarize the concrete answer or checklist inside the guide.
date / publishedAt article:published_time and JSON-LD datePublished ISO `YYYY-MM-DD` string; never change on routine typo fixes.
updatedAt Sitemap <lastmod> and JSON-LD dateModified Update only when substantive sections, code examples, or data tables are refreshed.
canonical (optional override) <link rel="canonical"> Defaults to `${SITE_URL}/blog/${slug}/`; override only for syndicated posts.
draft / noindex Excludes post from `sitemap.xml` and outputs `noindex` Boolean (`false` for published posts).
04 / MONOREPO CONTENT WORKFLOWS (PROMPT P17)

How an SEO app safely drafts and updates Markdown articles inside a GitHub monorepo

In a Turborepo, pnpm, or Nx monorepo, your marketing site and blog rarely sit at the root of the repository—they live inside a specific package such as `apps/site/src/guides` or `apps/web/content/blog`, alongside application code in `apps/desktop` and shared libraries in `packages/core`.

Letting an automated tool commit blindly to `main` in a monorepo is unacceptable. Here is how Searchbop isolates Markdown content workflows (`P17`):

Official Cloudflare Pages documentation & boundaries

Reference documentation: See Cloudflare Pages — Headers, Cloudflare Pages — Redirects, and Preview Deployments. Always verify your deployed headers using `curl -I https://your-project.pages.dev` and `curl -I https://yourdomain.com` after each release.

Research context & search queries covered on this page Next.js & Workflow · P1

Clusters & intent: nextjs & content-workflow (Informational / Commercial · P1) · Target queries: Cloudflare Pages SEO, Markdown blog SEO, AI article writer for Markdown, monorepo SEO tool, GitHub SEO automation.

Benchmark prompt addressed: P17 — Can an SEO app safely draft articles for a Markdown blog in a monorepo?