How Search Works in 2026: From Discovery to Generated Answers

Direct answer

Before a page can appear in Google, Google must discover its URL, crawl it, render any necessary JavaScript, understand and index the content, select a canonical URL, and decide that the page is relevant enough to serve. AI Overviews and AI Mode add retrieval, query fan-out, synthesis, and citations after those foundations. They do not give a broken or invisible page a special route into Google.

If your page is missing, do not begin by rewriting keywords or adding schema. Find the first failed stage in the pipeline. Fixing that first failure is the fastest reliable path forward.

I want you to leave this lesson able to do one thing well: look at a URL that is not performing and identify the next useful test, instead of guessing.

What you will be able to do

By the end, you will be able to:

• distinguish discovery, crawling, rendering, indexing, canonical selection, serving, snippets, AI retrieval, and citation;
• explain why indexable, indexed, ranking, mentioned, and cited are different states;
• diagnose a missing URL in the right order;
• avoid fixes that sound advanced but cannot solve the observed failure;
• complete a five-case search pipeline diagnosis with at least four correct root causes.

The model: Search is a pipeline

Google describes conventional Search in three broad stages: crawling, indexing, and serving results. Within an actual audit, that model becomes more useful when we separate discovery, fetching, rendering, canonical selection, ranking, and presentation.

Google's current How Search Works documentation says most pages are found automatically through crawling, that JavaScript may be rendered, and that crawling, indexing, and serving are not guaranteed. That last point matters. A valid page is eligible for consideration, not entitled to traffic.

Here is the practical sequence:

1. Discovery: a search system learns that a URL exists.
2. Crawl: a crawler requests the URL and receives an HTTP response.
3. Render: the system executes enough of the page to see content that depends on JavaScript.
4. Index: the system analyzes the content and decides whether to store it.
5. Canonical selection: similar URLs are clustered and one representative may be selected.
6. Serve and rank: the system matches indexed material to a query and orders useful results.
7. Present a snippet: the system chooses a title link, passage, image, or other presentation.
8. Retrieve for a generated answer: a generative system searches for supporting information, sometimes through multiple related searches.
9. Synthesize and cite: it produces an answer and may link to sources used as support.

The first seven stages explain why a normal blue link can or cannot appear. The last two help explain generative search surfaces. They are connected, but a citation is not simply “position zero.”

What each stage actually means

Discovery

There is no complete central registry of every public page. Crawlers find URLs through links, previously known URLs, redirects, and submitted sitemaps. A sitemap is useful discovery support, not a command to index everything in it.

A new lesson linked from the Academy hub has a discovery path. The same lesson accessible only after a user types into a client-side search box may have no stable crawlable link at all.

Question to answer: does Google know the exact canonical URL exists?

Evidence to collect: internal links, sitemap inclusion, Search Console discovery state, server log requests, and the URL Inspection result.

Crawl

Googlebot must be able to request the URL. A 200 response is not automatically good, and a blocked response is not automatically a deindexing instruction. Check the actual status, headers, redirect path, authentication, robots rules, and server stability.

Robots controls crawler access. They do not protect private data. If content must remain private, require authentication.

Question to answer: can the intended crawler fetch the intended public response?

Render

Modern Google Search can render JavaScript, but rendering is still a dependency. If the server sends an empty application shell and the primary copy arrives only after an API request, that request must work for the renderer. A browser screenshot on your laptop does not prove a crawler received the same content.

Compare three things:

• the HTTP response HTML;
• the rendered Document Object Model, or DOM;
• the network requests needed to create the rendered state.

If the main answer is missing from both response HTML and the rendered DOM, the page has no useful text for indexing regardless of how polished it looks after you log in.

Indexing and canonical selection

After crawling, Google analyzes text, images, video, titles, links, language, and other page signals. It can decline to index low-value or inaccessible material. It also groups substantially similar pages and may select a canonical representative.

A declared canonical is a signal, not an order. If your canonical tag points to URL A while internal links, redirects, and the sitemap consistently support URL B, you have created a conflict.

Indexable means no known rule prevents indexing.
Indexed means the system actually stored the page.
Canonical means the representative URL selected for a duplicate cluster.

Those terms are not interchangeable.

Serving and ranking

An indexed page still needs to be relevant and useful for a specific search in a specific context. Google documents many systems involved in ranking, but it does not publish a formula you can reproduce. Its ranking systems guide is a map of documented systems, not a checklist that guarantees a position.

Rankings can vary by query wording, language, location, device, freshness needs, and the shape of the results page. One manual search is an observation, not a market-wide measurement.

Snippets and eligibility

Google normally creates snippets from visible page content and can choose a different passage for different searches. A page can rank but offer weak extractable support for the exact question. It can also limit snippets through directives.

This distinction matters in generative results because Google says pages need to be indexed and eligible for a normal snippet to be considered as supporting links in AI features. The official AI features and your website documentation also says there are no additional technical requirements for appearing in AI Overviews or AI Mode.

Retrieval, query fan-out, and synthesis

A complex prompt may contain several decisions. Instead of performing one search, a system can issue related searches in parallel, retrieve useful material, and synthesize an answer. Google calls this query fan-out in its Generative AI optimization guide.

Imagine the request:

What is the best CRM for a 20-person plumbing company moving from spreadsheets, and how long will migration take?

A useful answer may need evidence about field-service workflows, mobile use, seat pricing, import formats, migration support, and implementation time. Those are plausible information needs. They are not a leaked list of hidden queries, and they do not justify six near-duplicate pages.

The system may cite a migration guide even when that guide would not rank for the full original prompt. That is why an AI citation does not prove the cited URL held a conventional ranking position for the original wording.

The complete diagnostic table

Use the table from top to bottom. A later-stage symptom does not erase an earlier failure.

A repeatable nine-step audit

1. Write the intended URL state

Record the exact URL, whether it should be public, its intended canonical, language, and the user task it completes. You cannot diagnose a page if the desired state is unclear.

2. Request the URL without a logged-in session

Record the final status, redirect chain, response headers, and whether the final page is the intended one. A public lesson should not depend on a cookie or account.

3. Check crawler controls

Review robots rules and page-level robots directives. Remember that blocking a crawl can prevent a crawler from reading a noindex directive on the page.

4. Compare response HTML and rendered DOM

Find the H1 and the direct answer in both. Record failed scripts or API calls. The important question is not “does JavaScript work?” It is “is the primary meaning reliably present?”

5. Reconcile canonical signals

Compare the HTML canonical, redirects, internal links, sitemap URL, hreflang references, and preferred protocol and hostname.

6. Inspect Google’s known state

Use URL Inspection for the indexed state and a live test for the current fetchable state. These can differ because the index reflects an earlier crawl.

7. Check performance by cohort

If indexed, segment page data by country, device, date, and relevant query class. Do not diagnose a French mobile visibility problem from a sitewide desktop average.

8. Inspect the result and answer surface

Record the query, locale, device, date, logged-in state, result type, displayed title, snippet, and citations. Treat this as a reproducible observation.

9. Fix the first failed stage and verify again

Choose the smallest complete fix. Record the before evidence, change, release time, and expected observation. Then wait for the relevant system to update before drawing a conclusion.

Worked example: the page that looks fine to the team

Suppose /academy/technical-seo-basics/ returns 200. It is allowed in robots, has a self-canonical, appears in the sitemap, and looks complete in a developer's logged-in browser. Yet URL Inspection reports little visible content.

The response HTML contains:

<main id="lesson"></main>
<script src="/lesson-app.js"></script>

The script requests /api/lessons/technical-seo-basics. That endpoint returns 401 without a session cookie. The developer has a session, so the page looks normal. A fresh visitor and a renderer without that cookie receive an empty lesson.

The first failed stage is rendering. The correct fix is to serve the public lesson content in the initial response or make the public content request reliably accessible, while keeping private account data behind authorization.

Adding Article schema would describe content that the renderer still cannot see. Adding llms.txt would not repair the failed request. Repeating an indexing request would only ask Google to process the same broken state again.

Why this can still fail after the fix

Once the copy renders, Google can crawl it and still choose not to index or rank it. The content may duplicate another canonical lesson, fail the task, or face stronger alternatives. Repairing rendering creates eligibility. It does not guarantee the later stages.

Common mistakes and what to do instead

“Robots.txt will remove the page”

Robots rules control crawling. Use the correct indexing or removal mechanism for public content, and authentication for private content.

“URL Inspection says it is indexable, so it is indexed”

Indexability is a possibility. Check the actual indexed state and selected canonical.

“It ranks for me, so the problem is solved”

Record an unbiased cohort. Your location, language, device, and search history can affect what you see.

“An AI answer cited us, so we rank for that prompt”

A generative answer may retrieve a source for one supporting claim. Track the citation separately from conventional position.

“The algorithm changed”

That can happen, but it is not the first diagnosis. Verify response, render, directives, canonical, and reporting before invoking a broad external cause.

SEOryon Search Pipeline Diagnostic

Download the Search Pipeline Diagnostic CSV. For each problem URL, add:

• the desired state;
• the observed symptom;
• the earliest suspected stage;
• one piece of supporting evidence;
• the next falsifiable test;
• the owner;
• the fix you will deliberately not try yet.

My rule is simple: every audit row needs evidence and a next test. “Traffic is down” is not a diagnosis. “Mobile impressions fell on the selected canonical after the rendered H1 disappeared” is.

Exercise: five URLs, five decisions

For each case, name the failed stage, the supporting observation, the next test, and one inappropriate fix.

1. A new article returns 200 and renders correctly, but no internal page or sitemap links to it. Inspection says Google does not know the URL.
2. A public guide is internally linked, but the server returns 503 to Googlebot during every live test.
3. A page is crawled. The response and rendered DOM contain noindex, although the CMS editor shows “index.”
4. Three identical URLs exist. Internal links and the sitemap point to URL B, while URL A declares itself canonical.
5. A page is indexed and earns impressions, but a monitored AI answer repeatedly cites a regulator for a legal limit that your page states without a source.

Answer key

1. Discovery. Link it from a crawlable relevant hub and include the canonical URL in the sitemap. Schema is not the first fix.
2. Crawl or server availability. Inspect logs, capacity, and response handling. Rewriting the title cannot help.
3. Indexing directive. Inspect the production HTML path and CMS rendering logic. An indexing request before removing the directive is wasteful.
4. Canonical conflict. Choose one owner and align canonical, links, redirects where appropriate, and sitemap. Buying links to both versions adds noise.
5. AI retrieval and citation support. Add primary evidence and precise scope if the statement belongs on the page. Creating a near-duplicate page for the prompt is inappropriate.

Score one point per correct failed stage and one per defensible next test. A strong pass is at least eight points out of ten, including four correct stages.

Final checklist

• The intended public and canonical state is written down.
• A clean request reaches the intended final URL.
• The response status and headers match that intent.
• Crawlers are not unintentionally blocked.
• The H1 and primary answer exist in the rendered DOM.
• Canonical, redirects, internal links, sitemap, and hreflang agree.
• URL Inspection distinguishes live state from indexed state.
• Performance is segmented by a relevant page, country, device, date, and query cohort.
• Manual result observations include query, locale, device, and date.
• AI mentions, citations, owned links, and referral visits are measured separately.
• The first failed stage has an owner and a verification test.
• No later-stage “optimization” is being used to avoid an earlier technical fix.

Frequently asked questions

How long does Google take to index a page?

There is no universal time. Discovery, crawl demand, site health, duplication, content value, and system scheduling all matter. Google does not guarantee indexing. Measure the actual state rather than promising a number.

Does submitting a sitemap make a page rank?

No. A sitemap can help discovery and reinforce canonical information. It does not guarantee crawling, indexing, or ranking.

Does Google execute JavaScript?

Google documents rendering JavaScript with a recent version of Chrome, but unreliable dependencies can still hide content. Other crawlers and assistants may have different capabilities. Server-delivered semantic content is the most robust baseline.

Do I need special schema or an llms.txt file for AI Overviews?

Google's July 2026 guidance says no special schema or AI file is required for Google Search. Use truthful structured data when it helps describe visible content, and repair ordinary crawl, render, index, and content problems first.

What is the difference between an AI mention and a citation?

A mention names your brand. A citation points to a source, which may be your site or a third party. A referral is a measurable visit from a link. Track all three separately.

Sources and methodology

This lesson prioritizes official Google documentation for product behavior. The diagnostic model is an SEOryon teaching framework that separates Google's broad stages into observable audit states. Community discussions were reviewed to identify beginner wording, including questions about how a search engine can process the scale of the web. Community posts were not used as factual authority.

1. Google Search Central, In-depth guide to how Google Search works, updated 18 December 2025. Official documentation. Google notes that crawling, indexing, and serving are not guaranteed.
2. Google Search Central, AI features and your website, updated 10 December 2025. Official documentation. Eligibility guidance does not guarantee inclusion.
3. Google Search Central, Top ways to ensure your content performs well in Google's generative AI experiences, updated 10 July 2026. Official documentation. It explains query fan-out without disclosing ranking weights.
4. Google Search Central, A guide to Google Search ranking systems, updated 10 December 2025. Official documentation. The guide names documented systems but is not a reproducible scoring formula.

Next: SEO, AEO, and GEO: Precise Definitions Without the Myths