Inside the Machine: What Actually Happens When You Ask an AI a Question

Picture a customer in Hamilton typing "best firm mattress for back pain under $1,500" into ChatGPT instead of Google. Four seconds later she has a paragraph naming three retailers, with little numbered links beside the claims. No results page. No scrolling. If your store is one of the three, you just received the highest-intent referral in commerce. If it is not, you never existed in that buying moment. Understanding why the machine picked those three is not trivia; it is the mechanics of where your next customer comes from. This page walks the pipeline end to end, in plain language, grounded in the actual research papers the industry built these systems from.

The pipeline in one sentence: fetch first, write second

Every major AI search product, ChatGPT Search, Perplexity, Google's AI Overviews, Microsoft Copilot, follows the same architecture: retrieve documents, then generate an answer from them. The research community calls it retrieval-augmented generation, or RAG, and the canonical formulation comes from Lewis and colleagues at Facebook AI Research in 2020 (arXiv:2005.11401). Their insight was that language models store a great deal of knowledge in their parameters but cannot easily revise it, explain it, or prove it, and they sometimes invent facts outright. Bolting a retriever onto the model fixes those weaknesses in one move: as the authors put it, "knowledge can be directly revised and expanded," and the evidence behind an answer becomes inspectable.

The paper's own evaluators found retrieval-grounded output more factual than a model writing from memory in 42.7 percent of head-to-head comparisons, against only 7.1 percent the other way. That gap is the entire reason your business can influence AI answers at all. A model answering purely from its training memory is a closed book you cannot edit. A model that fetches documents at answer time is reading the live web, and the live web is something you publish to.

So the honest framing for a business owner is this: AI search is a reading machine bolted to a writing machine. Your job is to be what the reading machine picks up.

Step one: how the engine finds candidate sources

The retrieval step does not match keywords the way 2010s Google did. It matches meaning. The technique, dense passage retrieval, was demonstrated by Karpukhin and colleagues in 2020 (arXiv:2004.04906): every passage on the web gets converted into a vector, a list of numbers encoding what the passage is about, and your question gets converted into a vector the same way. The engine then finds the passages whose vectors sit closest to the question's vector. Their example is worth repeating: asked "who is the bad guy in Lord of the Rings," a keyword system struggles because the answer passage says "villain Sauron," never "bad guy." A dense retriever maps both phrasings to nearly the same point in vector space and fetches the passage anyway.

The performance difference was not subtle. The dense approach beat the strongest keyword system of the era, in the authors' words, "by 9%-19% absolute in terms of top-20 passage retrieval accuracy." Production systems today typically run both methods together and re-rank the merged results, but the dense side is what lets an engine connect a customer's casual phrasing to your carefully written page.

Two practical consequences follow. First, you no longer win by repeating a keyword; you win by having a passage whose meaning genuinely answers the question. Second, retrieval happens at the passage level, roughly paragraph-sized chunks, not the page level. A 3,000-word page with no single paragraph that answers the question cleanly loses to a page with one tight, self-contained answer paragraph. This is Vector 4 in our methodology, Embed: write the exact passages the machines extract.

Step two: how the answer gets written

Once the retriever hands over its top passages, usually five to ten of them, the language model writes the answer conditioned on that bundle plus your question. The Lewis paper describes two ways of doing it: the model can lean on one retrieved document for the whole answer, or draw on a different document for different parts of the sentence it is building. Modern engines behave like the second mode. Watch a Perplexity answer assemble itself and you can see it: the pricing claim carries one citation, the durability claim another, the local availability claim a third. The answer is a weave, and each thread traces back to a specific source.

This weaving explains something owners find counterintuitive: you do not need to be the best page on the whole topic to appear in an answer. You need to be the best passage for one specific claim inside the answer. A Brantford retailer will rarely out-rank a national publication on "what is memory foam," but it can absolutely own "memory foam firmness for side sleepers in a cold Ontario bedroom" because nobody else wrote that passage at all.

The generation step is also where hallucination risk lives. When the retrieved bundle does not contain the fact the model needs, the model may fill the gap from its parametric memory, confidently and sometimes wrongly. Engines fight this by tightening the leash between claims and sources, which raises the value of pages that state facts plainly enough to be quoted without interpretation.

The knowledge the model already has, and where it runs out

Not every AI answer involves retrieval. Ask a model who wrote Hamlet and it answers from training memory, no lookup, no citations. The interesting question for a business is where memory runs out, because that boundary is where retrieval, and therefore your website, takes over.

The boundary sits at the long tail. Facts that appear in millions of training documents are baked into the model's parameters. Facts that appear in a handful of documents, which describes nearly everything about a local or niche business, are not reliably learned, and the model must fetch a source to answer questions about them. The Lewis team demonstrated the flip side of this dependence with an elegant experiment: they swapped the retrieval index from a 2016 snapshot of Wikipedia to a 2018 one and the model's answers about world leaders updated accordingly, no retraining required. The model's beliefs about the changeable world live in the index, not the weights.

Read as strategy: for questions about your niche, your prices, your service area, your product tradeoffs, the model is structurally forced to consult the web. Whoever has published the clearest retrievable passage on those questions gets to be the machine's memory. We unpack the business side of this architecture in our piece on what RAG means for business visibility.

Where the citations come from

Citations are not decoration; they are the pipeline showing its work. When an engine attributes a claim, it is pointing at the retrieved passage that grounded that span of text. Three things decide whether the pointer lands on you.

You have to be in the index. Each engine reads a different slice of the web: Google's AI features read Google's crawl, ChatGPT leans on Bing's index and its own fetching, Perplexity runs its own crawler. A robots.txt rule or a JavaScript-only page can quietly remove you from one engine's universe while leaving you visible in another. Our audit of where each AI engine actually reads from maps these surfaces engine by engine.

Your passage has to survive re-ranking. Being fetched puts you in a pool of dozens of candidates; only a few survive into the final answer. Trust signals decide the cut: consistency of your entity details across the web, agreement between your claims and other sources, structured data that removes ambiguity about who is speaking.

Rank is not the deciding vote. A 2026 study measuring Google AI Overviews across 55,393 queries (arXiv:2605.14021) found nearly 30 percent of cited domains did not appear on page one of the adjacent organic results. The Princeton group that coined generative engine optimization (Aggarwal et al., arXiv:2311.09735) found the same decoupling from the other direction: adding quotations, statistics, and citations to a page raised its visibility in generated answers by up to 40 percent in their benchmark, with no change in classic rank. We break that paper down in our GEO paper explainer.

Why the same question gets different answers

Ask an assistant the identical question twice and you will often get two different answers, sometimes citing different sources. Owners read this as randomness. It is actually three specific mechanisms stacking.

• The retrieval pool shifts. Indexes refresh continuously. A page crawled an hour ago can enter today's candidate pool and displace yesterday's source. Two runs minutes apart can draw from different pools.
• Generation is sampled, not computed. Language models choose each word probabilistically. Small early differences in wording cascade into different sentence structures, different emphasis, and different claims making the final cut, which changes which citations attach.
• The pipeline itself is a moving target. Engines run live experiments on retrieval depth, re-ranking weights, and answer formats. The system answering you on Tuesday is not byte-identical to Monday's.

In Matt's testing across client niches, this variance follows a pattern worth knowing: "When we track a prompt daily for a month, the brands that appear in ninety percent of runs are never the ones with one great page. They are the ones the retriever can grab from six or seven different directions. Thin presence produces coin-flip citations; dense presence produces stable ones." That observation is the practical answer to variance: you cannot stop the dice from rolling, but you can hold more of the sides.

Same skeleton, four different animals

The retrieve-then-generate skeleton is universal, but each engine dresses it differently, and the differences matter when you decide where to spend effort.

• Google AI Overviews trigger on a subset of queries, lean on Google's own index and ranking systems as the candidate pool, and by mid-2026 appear on roughly half of informational searches. Inclusion correlates with, but is not decided by, organic rank.
• ChatGPT Search decides per-question whether to retrieve at all. When it does, it fetches through Bing and its own crawler and cites inline. When it does not, it answers from memory and your website is irrelevant to that response.
• Perplexity retrieves on every query by design and is the most citation-forward of the group, typically attributing five or more sources per answer, which makes it the easiest engine to win a first citation on.
• Copilot and Gemini blend the patterns above with their own indexes and product surfaces.

One brand, four pipelines, four different verdicts on whether you exist. Any measurement of AI visibility that checks a single engine is measuring a quarter of the market.

What a business can influence at each stage

Here is the whole article compressed into an operator's table. Every row is a pipeline stage; every right-hand cell is work you can actually schedule.

The re-ranking row deserves one concrete example, because "schema markup" stays abstract until you see it. This is the kind of block that tells every engine, in machine-readable form, exactly which entity is speaking:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "Your Business Name",
  "address": {
    "@type": "PostalAddress",
    "addressLocality": "Brantford",
    "addressRegion": "ON",
    "addressCountry": "CA"
  },
  "telephone": "+1-519-555-0123",
  "url": "https://example.com"
}
</script>

Ten lines. It settles who you are, where you are, and which URL is canonical for you, three of the ambiguities that get candidate passages cut at re-ranking. This whole stage of work is Vector 6, Structure, in the Formative Digital research library, where the deeper technical material behind this page lives.

Why the pipeline favours the specific over the big

The pipeline's biases run opposite to the old search economy in one important way: it rewards specificity over sheer domain size, because retrieval operates on passages and long-tail questions have few competing passages. A national chain has ten thousand pages, but if none of them contains a passage answering the exact situational question a buyer asked, the retriever keeps looking.

Our clearest evidence is Mattress Miracle, an independent Brantford retailer whose organic presence grew from roughly 1,000 to 82,400 monthly visits (SEMrush, April 2026) on the strength of passage-level coverage of the questions mattress buyers actually ask machines. The standard qualifier applies and we mean it: results depend on industry, competition, and existing digital presence, and no pipeline knowledge turns into a guarantee. What the case does prove is that the mechanics described on this page respond to deliberate work at every stage, and that an independent business can do that work without a national budget.

If you want the wider discipline this page sits inside, start with what generative engine optimization is. If you want to know how your own site performs at each pipeline stage right now, the audit below checks all five rows of the table for you.

Prefer to walk through the pipeline with a person instead of a form? Get in touch and we will map your questions to the stages that matter for your business.