Technical SEO checklist before you scale content

The direct answer: fix discovery before you fix anything else

Before adding content volume, confirm that search engines can find, fetch, and index the pages you already have. A content calendar cannot compensate for URLs that are blocked, redirected incorrectly, duplicated, orphaned, or canonicalized to a different page — new articles just pile up behind the same technical wall. Work the checklist in this order: robots and sitemap, canonicals, redirects, rendering, internal links, metadata, structured data, and only then indexation versus discovery as the diagnostic that tells you whether the earlier steps actually worked.

The order matters because each layer depends on the ones before it being settled. Robots and sitemap decide whether a crawler is even permitted to try. Canonicals decide which version of a page a crawler should trust once it gets there. Redirects decide how much of a crawler's limited attention gets spent reaching the right URL versus bouncing through old ones. Rendering decides whether the content, canonical, and metadata a crawler eventually sees match what a human visitor sees. Internal links and metadata decide discoverability and presentation once the underlying page is technically sound, and structured data adds machine-readable context on top of content that already has to be correct on its own. Checking indexation versus discovery last is deliberate: it's the only item on this list that tells you whether the other eight actually worked, rather than whether they were merely attempted.

Robots and sitemap

Two files decide whether a crawler even attempts the rest of this list. `robots.txt` should allow the crawlers you want and disallow only what genuinely should not be indexed — an authenticated app, an internal tool, a staging path — not entire content sections by inherited default. Check that `llms.txt`, if you publish one, does not silently disallow the crawlers assistants use to answer questions — a rule copied from a template built for a different site can block exactly the traffic the file was meant to invite.

robots.txt controls crawling, not indexing, and the distinction matters more than most checklists imply. Google's own documentation on the file states plainly that it's used mainly to avoid overloading a site with requests, not as a mechanism for keeping a page out of Google — a URL disallowed in robots.txt can still be indexed without its content if another site links to it, which produces a search result with no description and no reliable way to fix it short of removing the block. If the actual goal is removing a page from search results, the correct tools are a noindex tag, a password wall, or removing the page outright, never a robots.txt disallow rule used as a substitute for those.

• Confirm robots.txt allows the paths that should rank and blocks only what should not
• Confirm the sitemap references the canonical URL for every page, not a redirect target or a parameter variant
• Confirm the sitemap returns HTTP 200 with the correct XML content type, not an HTML error page
• Submit the sitemap in Search Console and check the coverage report, not just that the file exists
• Remove retired or redirected URLs from the sitemap; a sitemap full of 3xx and 4xx responses reads as neglect
• Don't rely on a robots.txt disallow to deindex a page that's already indexed — Google can't see a noindex tag on a URL it's blocked from crawling, so the block alone can leave the URL indexed indefinitely
• For a comprehensively linked site under roughly 500 pages, a sitemap is optional rather than required — but it becomes useful again the moment the site is large, new with few external links, or carries rich media Google should associate with specific URLs

Canonicals

A canonical tag tells a search engine which URL is the authoritative version when several URLs could serve the same content — a product with tracking parameters, a paginated list, an HTTP and HTTPS pair, a www and non-www pair. Every page needs exactly one canonical, it needs to be absolute and self-referencing on the true version, and it needs to actually resolve to a 200 rather than a redirect or a 404.

Canonical preference is a hint, not a directive, and Google ranks the strength of the signals that inform its own choice: a permanent redirect is the strongest signal, a self-referencing `rel="canonical"` link is next, and inclusion in a sitemap is the weakest of the three — which is why a sitemap full of the right URLs can't fully compensate for a wrong or missing canonical tag on the page itself. The three signals stack, so a site that redirects duplicates, tags the survivor as canonical, and lists only the survivor in its sitemap gives Google far less room to disagree with the chosen URL than a site relying on just one of the three.

Google also applies its own default preferences on top of whatever a site declares: HTTPS is preferred over an equivalent HTTP page unless the HTTPS version has a bad certificate, insecure dependencies, or redirects back to HTTP, and URLs inside a reciprocal `hreflang` cluster are preferred as canonicals over URLs the cluster doesn't include. Neither preference can be overridden by wishful configuration — the fix for an HTTP page winning canonical status is almost always a certificate or redirect problem, not a stronger canonical tag.

On a JavaScript-rendered page, set the canonical in the initial HTML rather than injecting it later. Google's JavaScript SEO guidance is specific: if a canonical must be set with JavaScript, never change it to a value different from what the original HTML already declared, and if the HTML can't declare one at all, leave it out of the HTML entirely rather than let the two disagree during the render queue's delay.

• Every indexable page declares a canonical, including the canonical page itself
• Canonicals are absolute URLs, not relative paths that can resolve differently by context
• Parameter variants (sort, filter, session, tracking) canonicalize to the clean URL
• No canonical points at a URL that itself redirects or 404s
• Paginated series either canonicalize sensibly or are excluded from indexing, not both attempted at once
• Redirects, canonical tags, and sitemap entries all point to the same URL — mixed signals are a documented cause of Google overriding a declared canonical
• The HTTPS version of a page wins canonical status by default; fix certificate and redirect problems rather than fighting this preference with markup

Redirects

A redirect that works in a browser can still be wrong for a crawler. Chains — a URL that redirects to a URL that redirects again — waste crawl budget and can drop link equity at each hop. Loops stop a crawler outright. And a 302 used where a permanent move actually happened tells search engines the old URL might come back, which delays the transfer of ranking signal to the new one.

Redirect chains cost more than user-perceived latency. Google's own crawl-budget guidance lists avoiding long redirect chains as a direct lever on how efficiently a site's allotted crawling time gets spent, and it also treats redirects as the single strongest signal in canonicalization — stronger than a `rel="canonical"` tag or a sitemap entry — which is exactly why an accidental redirect pointed at the wrong URL, or a loop, does more damage than an accidental canonical tag would.

For a URL that's genuinely gone rather than moved, a 410 or 404 is more useful to a crawler than a redirect to the homepage: a real not-found status tells Google to stop trying that URL, while a homepage redirect just adds a hop to a URL Google will keep retrying regardless. Save redirects for URLs that actually have a destination; use a clean 404 or 410 for the ones that don't.

• Collapse redirect chains to a single hop from old URL to final destination
• Use 301 (permanent) for URLs that moved for good; reserve 302 for genuinely temporary moves
• Check for redirect loops, especially after a domain, trailing-slash, or www change
• Verify old URLs still referenced in backlinks or bookmarks redirect somewhere relevant, not to a generic homepage
• Return a real 404 or 410 for content that's genuinely gone rather than redirecting it to the homepage — a status code says stop trying; a homepage redirect just adds a hop

Rendering

Source HTML alone is not enough for a JavaScript-heavy site. What matters is what a crawler sees after rendering: the title, description, headings, body content, links, structured data, and canonical as they exist in the rendered DOM, not just the initial server response. Compare the two directly rather than assuming a framework's default behavior is sufficient — a component that injects the canonical or the main content client-side can be invisible to a crawler that does not execute enough JavaScript, or executes it on a budget that runs out before your content appears.

Google processes a JavaScript-dependent page in three phases — crawling, rendering, and indexing — and the two queues involved aren't synchronized: a URL that passes robots.txt gets crawled, but it can then sit in the render queue for anywhere from a few seconds to considerably longer before a headless Chromium instance actually executes the JavaScript and the content becomes visible to indexing. Anything that only exists after that render step — an injected canonical, a client-rendered title, a component that fetches its own copy of the content — is invisible to Google until the render queue clears, which is exactly the gap server-side rendering and static generation exist to close.

Two narrower rendering traps are worth checking directly. First, Google can only follow links that are real HTML anchor elements with an `href` attribute — a click handler with no `href`, or a route driven purely by a URL fragment instead of the History API, is invisible to link discovery regardless of how the rendered page looks to a person. Second, if a component uses a shadow DOM, confirm the rendered output actually exposes the projected content; Google flattens light and shadow DOM when it renders a page, but only what ends up in the rendered HTML is indexable, so a poorly composed web component can hide content that looks fine in a browser.

• Fetch the rendered HTML, not just the initial response, and diff it against the source
• Confirm critical content (price, availability, the answer to the page's main question) exists in rendered HTML, not only after a client-side data fetch
• Confirm the canonical tag is present in both server and rendered output and they agree
• Check Core Web Vitals for the rendered experience, since render-blocking scripts affect both users and crawl efficiency
• Confirm every link a crawler needs to follow is a real anchor tag with an href, not a fragment-only route or a click handler with no href
• For components using shadow DOM, verify the rendered HTML actually contains the projected content — check with the Rich Results Test or the URL Inspection tool's rendered-HTML view, not just a browser

Internal links

A page with no internal links pointing to it is orphaned even if it is in the sitemap — sitemaps are a hint, not a guarantee, and pages a crawler can reach by following links get discovered and re-crawled more reliably than pages it can only find in an XML file.

Sitemap inclusion is the weakest of Google's canonicalization and discovery signals for a reason: a sitemap is a claim the site makes about itself, while a link followed from an already-crawled page is evidence the crawler collected directly. That's why a page can sit in a sitemap indefinitely without ever being crawled if nothing else on the site links to it — the sitemap is a hint that a URL exists, not a guarantee it will be visited.

`nofollow` on a link and no `href` at all have the same practical effect on discovery even though they mean different things: neither passes the crawler forward. Use `nofollow` deliberately, on links that genuinely shouldn't pass crawl priority — a login link, a sponsored placement — not as a default applied to entire navigation regions, which can quietly orphan everything reachable only through them.

• Every priority page has at least one contextual internal link from a page that already gets crawled
• Navigation and footer links use real anchor tags with resolvable hrefs, not JavaScript-only click handlers
• Broken internal links are fixed or removed, not left to return a 404
• Anchor text describes the destination rather than repeating an exact-match keyword on every link
• Don't apply nofollow to entire navigation regions by default — check that priority pages still have at least one followable path from an already-crawled page

Metadata

Titles and descriptions are the smallest item on this list and the one most likely to be templated into uselessness. A title that is the same pattern with one word swapped across a thousand pages gives a search engine nothing to differentiate on, and a description that does not match what the page actually answers gets rewritten by the search engine anyway.

Google is explicit that title-link generation is fully automated: it draws on the `<title>` element, but also on the page's main visual heading, `<h1>` and other heading elements, `og:title`, other large or stylistically prominent text, anchor text pointing at the page, and even WebSite structured data — and when those sources disagree, Google picks whichever one it judges best represents the page for that specific query. A `<title>` element that matches the page's H1 in substance, even if not word-for-word, gives Google far less reason to substitute something else.

Google's documentation names specific patterns that trigger a rewritten title link: a half-empty title missing part of the intended text; an obsolete title left over from a template that wasn't updated (a page that visibly says "2027 admissions" while the title still says "2026"); micro-boilerplate that's identical across a set of pages except for one buried variable, like a season number or a location; no single element on the page that clearly reads as the main heading, when several compete at the same visual weight; and a language or script mismatch, where the title is transliterated into a different alphabet than the page's actual content. Auditing metadata means checking for these patterns specifically, not just confirming a title tag exists.

• Every indexable page has a unique title that states what the page is, not a boilerplate pattern
• Descriptions summarize the specific answer or offer on the page, not a generic company line
• Heading structure (one H1, logically nested H2s and below) matches the visible content order
• No duplicate titles or descriptions across pages meant to rank for different queries
• Confirm the title element and the page's main visible heading agree in substance — a mismatch is one of Google's documented triggers for a rewritten title link
• Avoid micro-boilerplate: a shared title pattern across a page series must vary its buried detail (a season, a date, a size) in a way that's actually distinct, not just present

Structured data

Schema should describe what a reader can actually see on the page. Organization, product, article, author, breadcrumb, and FAQ markup can clarify relationships for both search engines and assistants deciding whether a passage is attributable — but markup that claims more than the visible content supports creates ambiguity a validator will not catch, because it checks syntax, not truthfulness.

Google's general structured-data guidelines separate two kinds of failure. A technical failure — wrong format, a missing required property, markup blocked from Googlebot by robots.txt or noindex — is what the Rich Results Test and the URL Inspection tool catch, and it simply makes a page ineligible for a rich result. A quality failure — content invisible to readers, markup that misrepresents the page, fake reviews, structured data placed on a page that doesn't actually contain what it describes — isn't caught by any validator, can result in a manual action in Search Console, and is judged on intent rather than syntax.

Two placement rules are easy to get wrong on a real site. Structured data belongs on the page it describes, not on a summary page that only links to detail pages elsewhere — a category page shouldn't carry Product schema for products it merely links to. And on a specific type, use the most specific applicable schema.org type rather than a generic one; a page about a single article should use Article or a more specific subtype rather than a bare CreativeWork, since Google's documentation treats specificity as part of the quality bar, not a stylistic preference.

• Validate every structured-data type against the current schema.org and Google guidelines, not just that it parses
• Confirm each schema field matches visible page content — no FAQ schema for questions that are not visibly answered on the page
• Keep Organization and author identity consistent across every page that references them
• Re-validate after any template change; one component edit can silently break markup sitewide
• Structured data goes on the page it describes, not a summary or category page that only links to the detail pages
• Use the most specific schema.org type that applies, not a generic parent type — Google's guidelines treat specificity as part of eligibility

Indexation versus discovery

These are two different states and Search Console reports them separately for a reason. A discovered page is one a crawler found a link to or read in the sitemap. An indexed page is one the search engine decided was worth adding to its index. A page can be discovered and sit in "Crawled — currently not indexed" indefinitely, which is a quality or duplication signal, not a technical error — the earlier items on this list rule out the technical causes so that a discovered-but-unindexed page is left with quality as the remaining explanation.

Search Console's own definitions are precise enough to quote directly. "Discovered — currently not indexed" means the page was found by Google but not crawled yet — typically because Google expected crawling it right away would overload the site, so the crawl was rescheduled, which is why the last-crawl date on that status is always empty. "Crawled — currently not indexed" means Google did fetch the page and chose not to add it to the index anyway; there's no promise it ever will be, and no need to resubmit the URL for crawling, because resubmission doesn't change the underlying evaluation.

Check the coverage report by status, not just the total indexed count: "Discovered — currently not indexed" often means crawl budget or internal linking; "Crawled — currently not indexed" more often means the page reads as thin or duplicative once a human or a search engine actually reads it.

Crawl budget: when it's actually the bottleneck

Crawl budget gets invoked as an explanation far more often than it's actually the constraint. Google's own guidance scopes it narrowly: it matters for sites with a million or more unique pages that change at least weekly, sites with ten thousand or more pages that change daily, or any site where a large share of total URLs sit in "Discovered — currently not indexed." Below that scale, and assuming the rest of this checklist is in order, crawl budget is rarely what's holding a site back — content quality, internal linking, and the technical items above account for most indexing shortfalls on small and mid-size sites.

For a site that does clear that threshold, budget is the product of two separate levers, and they need different fixes. Crawl capacity is about server health: response time, latency, and error rate determine how many parallel connections Google is willing to hold open, and a site that responds slower or throws more 5xx and 429 errors gets that ceiling lowered automatically. Crawl demand is about content: how much of a site's URL space Google perceives as worth visiting, weighted by popularity and by how often pages actually change. The most direct way to raise demand without waiting for popularity to grow is to shrink the URL space Google has to consider — consolidating duplicate content and blocking unimportant URL variants in robots.txt both reduce how much of a crawler's attention gets spent on pages that were never going to rank anyway.

• Keep average server response time low and error rates near zero — Google's crawl capacity limit adjusts automatically based on how a site's server behaves under crawling load
• Consolidate or block duplicate and low-value URL variants (parameters, infinite scroll, sort and filter combinations) rather than letting Google spend budget discovering all of them
• Support 304 Not Modified responses where possible so an unchanged page doesn't cost a full re-crawl
• Treat crawl-budget optimization as a large-site concern; smaller sites with the checklist above in order rarely need it

Verify, don't assume: how to check each item is actually true

Every item above has a way to check it directly rather than infer it from a dashboard summary. Fetch a page's rendered HTML through the URL Inspection tool or the Rich Results Test and compare it against the raw source — that's the fastest way to catch a canonical, title, or piece of structured data that only exists before or after JavaScript runs, rather than in both. A `curl -I` request against a URL shows the actual response headers a crawler receives, including redirect chains and any `X-Robots-Tag`, without a browser's rendering quietly papering over a problem a script would hit.

Search Console's own reports are more reliable than assuming a fix worked. The Page Indexing report shows real counts by status, not an estimate; the Crawl Stats report shows what Googlebot actually requested and how the server responded, which is the right place to confirm a server-health theory about crawl budget rather than guess at it; and a live URL Inspection test shows Google's own selected canonical next to whatever the page declares, which is the direct way to catch a case where Google chose to disagree.

• Diff rendered HTML against raw source for any page relying on client-side rendering, using the URL Inspection tool or the Rich Results Test
• Run curl -I against key URLs to see actual response codes, redirect targets, and headers a crawler receives
• Use the live URL Inspection test to compare the user-declared canonical against Google's selected canonical, not just whether a canonical tag exists
• Re-check after every template or plugin change — verification is a recurring habit, not a one-time audit step

How Foliora runs this checklist automatically

This is one section of the checklist, not the whole point of the page: Foliora re-reads the rendered version of a page after every change it publishes, rather than trusting an API success response. It compares the intended diff — the canonical, the schema, the heading, the internal link — against what the live page actually returned, and treats a mismatch as a failed change rather than a shipped one. That verification step exists because every item above can regress silently after a template update, a plugin change, or a redeploy, and the only reliable way to catch that is to check the rendered output again, not to assume the last audit still holds.

Common questions