Quick Answer: To audit your website for AI search readiness, run four manual checks in an afternoon with no paid tools: can AI crawlers reach your pages, does every engine read the same business, what do live prompts return, and can an engine lift your answer cleanly. Crawler access always comes first.

An AI search readiness audit answers one blunt question: when a buyer asks ChatGPT, Perplexity, Gemini, or Google's AI Overviews about what you do, can the engine find you, identify you correctly, and quote you. The GEO research paper by Aggarwal and colleagues, presented at KDD 2024, measured that structuring content for these engines can raise visibility in generated answers by up to 40%, which tells you the gap between "cited" and "invisible" is an engineering problem, not luck. This page is the manual version of that audit, the one an owner can run without hiring anyone.

Most published audit guides assume you will buy a monitoring platform or hand the work to an agency. You need neither for a reliable first read. What you need is a browser, your own robots.txt, the free tier of four AI engines, and roughly two to four hours. At the end you will hold a score for each of four areas and one prioritized fix list, ordered by whatever is costing you the most visibility right now. If you would rather have it done for you, our AI visibility audit runs the same logic across engines and benchmarks your competitors; this guide is written so you never have to.

Reading time: about 13 minutes. Doing time: an afternoon.

What AI search readiness actually measures

AI search readiness measures whether generative engines can complete three jobs against your site: retrieve the page, resolve which business it belongs to, and reuse a passage in an answer. A traditional SEO audit stops at the first job and asks whether you rank in the ten blue links. That is why your existing audit misses this: ranking well in classic search and being quoted in an AI answer now overlap far less than they used to, so a page can rank on page one and still never appear in a single generated response.

The four checks below map to those jobs plus the measurement that ties them together. Retrieval is crawler access. Resolution is entity consistency. Reuse is extractability. And the prompt battery is how you observe all three from the outside, the way the engines actually expose them. You do not need to instrument anything. Open a browser, keep a simple scorecard beside you, and read what comes back. The only prerequisites are your website URL, your business details, the names of two or three competitors, and a logged-out browser window so your own history does not colour the results.

Check 1: Can AI crawlers reach your pages?

Start here because an engine that cannot fetch a page cannot cite it, which makes crawler access the gate every other check depends on. Open your robots.txt directly in a browser by typing your domain followed by /robots.txt, then read it for any rule that blocks the AI user agents. The failure is rarely deliberate. It is usually a blanket User-agent: * Disallow: / left over from a staging site, or a security plugin that added AI bots to a block list alongside genuine scrapers.

These are the crawler tokens worth confirming by name. Each is controlled by its own User-agent block in robots.txt, and each does a different job:

User agent Operator What it does
GPTBot OpenAI Fetches pages that may inform ChatGPT's models
OAI-SearchBot OpenAI Surfaces sites inside ChatGPT search results
ClaudeBot Anthropic Crawls pages for Claude
PerplexityBot Perplexity Indexes pages for Perplexity answers
Google-Extended Google Controls Gemini and AI grounding use of your content

You do not have to write anything to permit these agents, because a robots.txt with no matching Disallow already allows them. Writing the permission out anyway documents the intent, so the next person who tidies the file does not fold your AI bots into a block by accident. A minimal explicit allow looks like this:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Two traps hide behind a clean robots.txt. The first is your CDN or firewall: Cloudflare and similar providers now ship a one-click "block AI bots" control, and a well-meaning colleague can switch it on without touching the site. Check that dashboard, not just the file. The second is rendering. Load a few key pages with JavaScript disabled in your browser, or view the raw page source, and confirm the main content is still there. If the article body vanishes without JavaScript, an engine reading the raw HTML sees an empty shell, and server-side rendering or prerendering becomes the fix. Confirming access is the first move in our methodology, the diagnostic step we call Vector 1.

One tactic you can cross off the list is the llms.txt file. It has been sold hard through 2026 as an AI-readiness must-have, and both of the top-ranking guides for this very topic still recommend it. Google's own documentation is direct: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." John Mueller reinforced it in June 2026, calling the file purely speculative, noting that no AI system uses it despite years of availability, and comparing it to the discredited keywords meta tag. His actual priority advice matches this check exactly: the most fundamental thing you can do is not block the AI agents. Spend the afternoon on robots.txt and your CDN, not on a markdown file no engine reads.

Check 2: Does every engine read the same business?

The second check confirms that the sources these engines read describe one consistent business, because an AI system assembles your identity from scattered fragments and contradictory fragments produce a blurred entity that earns nobody's citation. The fastest test is direct. In each engine, ask a plain question about your company, something like "Tell me about [your business name] in [your city]," and read the answer back critically. Wrong address, outdated phone number, a service you dropped years ago, or worst of all a confident merge with a similarly named company are all entity problems, and no amount of new blog content fixes them until the machine knows who you are.

Behind that test sits the mechanical work, what our framework calls Vector 2, Anchor. Your name, address, and phone number should be character-for-character identical across your website, your Google Business Profile, and the directories that rank when someone searches your name. Your founder or author name should be spelled the same on the site, on LinkedIn, and on any guest article. Organization and Person schema should point at those same profiles through sameAs properties so the engines can connect the fragments to one node. If you carry a Wikidata or Wikipedia entry, verify it is accurate; a correct Wikidata item is a useful disambiguation signal that propagates into AI systems over time, though it is a clarity tool, not a substitute for genuine authority, and inventing one to game the check is a waste of an afternoon. When the engines describe you wrongly and you cannot see why, our companion guide on why your business does not show up in AI search walks through the ranked causes.

Check 3: Run a prompt battery and record the answers

The third check is a prompt battery: ask the engines the questions your buyers actually type, and write down exactly what comes back. This is the heart of the audit and where most of your afternoon goes. Build ten to twenty prompts in three groups, so you can see the difference between being discoverable, being described, and being compared.

Three groups of prompts to run

  • Discovery prompts (no brand name): "best [service] in [city]," "who should I hire for [problem] near [city]," "recommend a [category] company in [region]." These reveal whether you exist to the engine when nobody names you.
  • Brand-aware prompts: "Tell me about [your business]," "is [your business] any good," "what does [your business] specialise in." These reveal whether the engine's picture of you is accurate.
  • Comparison prompts: "[your business] versus [competitor]," "alternatives to [competitor]." These reveal who the engine reaches for in your category and how you stack up.

Run every prompt across all four surfaces: ChatGPT, Perplexity, Gemini, and Google itself, where you read the AI Overview at the top of the results. Free tiers are enough for a first pass. For each prompt, record four columns in your scorecard: are you named, are you cited with a clickable link, is the description accurate, and which domains did the engine cite instead of or alongside you. That last column is the most valuable thing the whole audit produces, because those cited domains are the sources you now need to appear on, contribute to, or out-publish.

Be honest about the method's limits. These models are non-deterministic, so run each prompt two or three times and note the range rather than trusting a single reply. Location and account history skew answers, which is why the logged-out window matters. Treat the result as a snapshot of one moment, not a fixed ranking you can bank. If you want the exact prompt set we use, our free AI visibility checker generates the discovery, brand, and comparison prompts for your business so you are not inventing them from scratch.

Matt Griffin, Formative Digital: "When I audit an Ontario service business, the prompt test tells me more in ten minutes than a stack of keyword reports. If all four engines describe you wrong, or name a competitor when nobody in the prompt did, you do not have a ranking problem, you have an identity problem. Fix who the machine thinks you are before you write another word of content."

Rather see your prompt results side by side?

Send us your domain and we will run the discovery, brand, and comparison prompts across ChatGPT, Perplexity, Gemini, and Google AI Overviews, then show you where you are cited, where you are missing, and which sources the engines trust instead. No charge, and a reply within one business day.

Check 4: Can an engine lift your answer cleanly?

The fourth check is extractability: whether an engine can pull a correct, self-contained answer off your page without having to stitch fragments together. Generative engines retrieve at the passage level, so they reward pages where a section stands on its own and punish pages where the answer is spread across three scrolls. Open your five most important pages and read each one the way a machine would, asking whether the answer is present, complete, and clearly attributed to a subject.

Score a page as extractable when it does the following. It states the answer in the first hundred to a hundred and fifty words, before the origin story. Its headings are shaped like the questions people ask, and each section re-states its subject by name instead of leaning on "it" or "this." Comparisons and numbers live in tables and lists rather than dense paragraphs. And the claims carry specifics: a named source, a dated statistic, a real quotation. The GEO study found that adding cited sources, quotations, and statistics were among the strongest levers for getting pulled into a generated answer, which means the same specifics that make a claim trustworthy are exactly what an engine reaches for when it needs something to quote.

Two structural notes close this check. Position matters, because engines weight the top and bottom of a document more heavily than the middle, a pattern we cover in our research piece on where AI engines read on a page, so put your answer where it will actually be read. And schema is worth adding even though it is not required: Google states plainly that there is "no special schema.org structured data that you need to add" for AI features, yet valid Organization, Article, and FAQ markup still helps machines connect your page to a verified entity, which is the machine-readability job of Vector 6, Structure. Add it as cheap insurance, not as a magic input.

Score it and build your fix list

Now convert four impressions into one prioritized plan by scoring each check from 0 to 3, then letting the numbers point at your first job. A zero means the check is failing or absent, a three means it is clean, and one or two cover the partial middle. Fill in the right-hand column as you go.

Check Score 0 (failing) Score 3 (clean) Yours
1. Crawler access AI agents blocked in robots.txt or CDN, or content needs JavaScript to appear All named agents allowed, CDN clear, content in raw HTML  
2. Entity consistency Engines describe you wrongly or confuse you with another company Matching NAP everywhere, correct descriptions, sameAs schema in place  
3. Prompt presence Never named or cited on discovery or comparison prompts Named and linked across engines on relevant prompts  
4. Extractability Answers buried, no self-contained sections, claims unsourced Answer-first, question headings, sourced data in lists and tables  

Priority is not simply "fix the lowest number," because the checks are not equal. Read the scores in this order. If crawler access scored 0, that is your entire to-do list until it is fixed, since the other three cannot pay off while the engines cannot fetch the page. Once access is clean, take the lowest remaining score first, and when extractability and entity are tied, favour extractability, because a well-structured page also gives the entity signals somewhere clean to live. The prompt battery is not something you "fix" directly; it is your instrument. Re-run it a month after each change, because Google Search Console and GA4 will not show you AI-answer citations, so those recorded prompt results are the only before-and-after you get.

Reading your total out of 12

10 to 12: AI-ready. You are eligible for citation across engines; move to monthly monitoring and a quarterly refresh. 7 to 9: readable but under-cited, usually an extractability or entity gap; fix the lowest check. 4 to 6: partial visibility with structural work needed across two areas. 0 to 3: largely invisible to AI search; start at crawler access and rebuild upward.

Set expectations honestly on what the fixes buy you. Crawler access and extractability tend to move citation eligibility within a couple of weeks of the next crawl, because they are on-page and get re-read quickly. Entity and reputation signals accumulate over months, since they depend on other sites agreeing about you. And a clean 12 makes you eligible to be cited, not certain to be; the engines still choose among eligible sources, and how fast anything shifts depends on your market, your competitors, and where you started. That honesty is the point of scoring it yourself: you end the afternoon knowing precisely which of the four jobs is the weak link, and the fix list writes itself.

Frequently Asked Questions

How long does it take to audit a website for AI search readiness yourself?

For a small site, plan two to four hours. The crawler and extractability checks take under an hour combined, and most of the time goes into the prompt battery, since you are running ten to twenty prompts across four engines and recording each result. Larger sites take longer only because there are more key pages to spot-check.

Do I need paid tools to run an AI search readiness audit?

No. The whole manual audit runs on a web browser, your own robots.txt file, and the free tiers of ChatGPT, Perplexity, Gemini, and Google. Paid monitoring platforms help you track citations continuously over time, but they are not required for a reliable first read of where you stand today.

Should I add an llms.txt file to improve AI search visibility?

It is not a priority. Google's documentation states you do not need special machine-readable files to appear in its AI features, and John Mueller confirmed in June 2026 that no major AI system currently uses llms.txt, comparing it to the old keywords meta tag. Spend the time making sure you do not block AI crawlers instead.

Which AI crawler user agents should I allow in robots.txt?

At minimum allow GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, and Google-Extended from Google. The common failure is a blanket rule that blocks everything, or a CDN bot-protection setting that blocks AI crawlers even when robots.txt allows them, so check both.

How is an AI search readiness audit different from a normal SEO audit?

A traditional SEO audit checks whether you rank in the ten blue links. An AI readiness audit checks whether four different engines can fetch your page, identify your business correctly, and quote you in a generated answer. The crawler tokens, the entity checks, and the prompt battery have no equivalent in a classic ranking audit.

How often should I repeat the audit?

Re-run the prompt battery monthly if you are actively competing for AI citations, and run the full four-check audit quarterly. Always repeat it after a site migration, a redesign, or any change to robots.txt or your CDN, since those are the changes most likely to cut off crawler access without warning.

Sources

  1. Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2024). GEO: Generative Engine Optimization. Proceedings of KDD 2024 (arXiv:2311.09735). Link
  2. Google (accessed 2026-07-20). AI features and your website. Google Search Central documentation. Link
  3. OpenAI (accessed 2026-07-20). OpenAI bots and crawler user agents. OpenAI developer documentation. Link
  4. Montti, R. (2026, June 2). Google Says LLMs.txt Is Purely Speculative For Now. Search Engine Journal. Link

Score your site and want a second read?

Formative Digital, Brantford, Ontario

You can run every check on this page without ever contacting us, and we built it that way on purpose. If you would rather compare your afternoon's score against what our agent team measures across the engines, send your domain and we will map the four checks for you, no cost and no retainer to see the results.

Request Your Free AI Visibility Audit