Skip to content

How to prepare your website for AI search: a checklist

15 technical checks so that Google AI Overviews, ChatGPT search, Perplexity, Gemini and Copilot can find and cite your website: robots.txt, indexing, HTML without JavaScript, page structure, schema.org and llms.txt.

For Google AI Overviews and AI Mode, ChatGPT search, Perplexity, Gemini or Copilot to find and cite your website, it first has to be technically accessible: AI search crawlers are not blocked, your key pages are in the Google and Bing indexes, and the main text is visible in the HTML without JavaScript. The second step is a clear page structure, visible dates, an author and consistent business details everywhere. Below is a 15-point checklist with explanations and links to each search engine’s official documentation.

This checklist covers technical readiness. How to write content and why mentions on other websites matter is explained in our article “How to get found in ChatGPT and Google AI answers”.

The checklist: what should you check?

Start with points 1–8: if a crawler cannot reach a page or the page is not indexed, structure and markup will not help.

No. What to check How to check Where it matters
1 OAI-SearchBot is not blocked open yoursite.lt/robots.txt ChatGPT search
2 PerplexityBot is not blocked robots.txt Perplexity
3 Claude-SearchBot and Claude-User are not blocked robots.txt Claude
4 Googlebot and Bingbot can reach key sections, CSS and JS robots.txt, URL Inspection in Search Console Google, Bing, Copilot
5 Key pages are in Google’s index Search Console > URL Inspection AI Overviews, AI Mode, Gemini
6 Site verified in Bing, sitemap.xml submitted Bing Webmaster Tools Bing, Copilot
7 No unnecessary noindex or nosnippet tags page source, URL Inspection Google AI features
8 Main text, prices and contact details are in the HTML “View page source” in your browser all AI crawlers
9 A short answer at the top of the page read the first paragraph all engines
10 Headings phrased as questions, comparisons in tables review your 5 most important pages all engines
11 Organization and Article markup without errors Rich Results Test Google
12 Visible publication and update dates the page and datePublished / dateModified Google
13 An author or responsible team is named article pages all engines
14 Consistent business name, address and phone number website, profiles, directories all engines
15 LCP, INP and CLS in the good range PageSpeed Insights Google, visitors

Can AI search crawlers reach your website?

Open yoursite.lt/robots.txt and check that there are no “Disallow” rules for AI search crawlers. Most companies separate their search crawlers from their model-training crawlers, so you can opt out of training without losing visibility in search.

Crawler Company Used for What blocking means
OAI-SearchBot OpenAI surfacing websites in ChatGPT search pages are not shown in ChatGPT search answers (navigational links may remain)
GPTBot OpenAI content for model training no effect on ChatGPT search
ChatGPT-User OpenAI opening a page at a user’s request robots.txt rules may not apply
PerplexityBot Perplexity surfacing and linking websites in Perplexity search; not used for training your site drops out of Perplexity search results
Claude-SearchBot Anthropic improving search result quality visibility in Claude search may fall
ClaudeBot Anthropic content for model training signals that content should be excluded from training
Googlebot Google Google Search, including AI Overviews and AI Mode pages disappear from Search and its AI features
Google-Extended Google not a separate crawler but a token: whether content may be used for Gemini models and grounding their answers no effect on inclusion or ranking in Google Search
Bingbot Microsoft Bing search pages are not added to Bing’s index

Sources: OpenAI, Perplexity, Anthropic, Google, Bing.

An example that blocks training but allows search:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.lt/sitemap.xml
  • Only block Google-Extended deliberately. According to Google, the token also covers grounding in the Gemini apps, where content from the Search index is supplied to the model.
  • Firewall. If your server or CDN has bot protection switched on, check that it does not block the search crawlers you allow in robots.txt.
  • User-request agents (ChatGPT-User, Perplexity-User, Claude-User) open a page when someone asks for it in a chat, so they treat robots.txt differently. See each company’s documentation.

Are your pages in the Google and Bing indexes?

Google AI Overviews and AI Mode have no extra requirements: Google Search Central says a page must be indexed and eligible to appear in Search with a snippet. So the first check is ordinary indexing.

Indexing checks: URL Inspection, robots.txt, noindex tag, sitemap.xml, canonical URLs and a single site address
Six indexing checks that also determine your visibility in AI search.
  • Google. Check your home page and 3–5 key pages with the URL Inspection tool in Search Console. The nosnippet, data-nosnippet, max-snippet and noindex controls limit what can be shown from a page, including in AI features.
  • Gemini. Because content from Google’s Search index can be supplied to Gemini for its answers, the same indexing matters here too.
  • Bing and Copilot. Verify your site in Bing Webmaster Tools and submit your sitemap.xml. Since February 2026 it has included the AI Performance report, which shows which of your pages are cited in Microsoft Copilot, Bing AI summaries and selected partner integrations.
  • IndexNow. Bing offers this protocol so that participating search engines learn about new, updated or removed pages straight away.

Is your text visible without JavaScript?

Your main text, prices and contact details should be in the HTML the server returns. Google renders JavaScript in a separate step after the page waits in a queue, and it notes that not all bots can run JavaScript, and that server-side rendering or pre-rendering is still a good idea. Other AI crawlers’ documentation does not say whether they execute JavaScript, so it is safest not to depend on it.

How to check: open the page, press Ctrl+U (“View page source”) and search for a sentence from the main text. If it is missing and you only see an empty container and script links, an AI crawler probably sees the same.

How should you structure a page so it is easy to cite?

Google says AI features need no special optimisation: the same principles of helpful, people-first content apply. In practice that means a clear structure that both people and search engines can read easily.

  • A short answer first. The first paragraph should answer the main question in 2–4 sentences, with the detail below.
  • Headings as questions. “How much does … cost?”, “How long does … take?”, “What is the difference between …?”, the way your customers ask.
  • Tables. Prices, timeframes, comparisons and specifications belong in a table, not scattered through the text.
  • One page, one topic. If the same service is described on three pages, it is unclear which one to cite.
  • Specific facts. A “from” price, a timeframe, an address and the steps of the work instead of generic phrases.

Do dates and authorship matter?

Google recommends showing a visible date labelled “Published” or “Updated” and using the same date in structured data (datePublished, dateModified). At the same time, Google lists changing a page’s date without substantial content changes as a sign of search-engine-first content. In the same guidance, Google encourages accurate authorship information, such as a byline linking to details about the author or team.

Are your business details consistent everywhere?

Your business name, address, phone number (NAP) and company code should match on your website, business profiles, company directories and social media. If different places show different addresses or an old phone number, outdated details may end up in an AI answer. Check your contact page, footer, structured data and at least 3–5 external profiles.

Which structured data do you need?

No special schema.org markup is needed for AI features; Google says so directly. Standard markup still helps a search engine understand your organisation and your articles, and the key rule is that markup must only describe what is visible on the page.

Type Where to use it What to include
Organization (or LocalBusiness) home page name, logo, address, contact details, sameAs links to your profiles
Article articles headline, author, datePublished, dateModified, image
BreadcrumbList all inner pages breadcrumb trail
FAQPage pages with an FAQ block only the questions and answers visible on the page

According to Google, Organization markup helps distinguish your business from other organisations with similar names. An important caveat about FAQPage: in September 2023 Google limited FAQ rich results to well-known government and health websites, and from 7 May 2026 stopped showing them in Search altogether (documentation updates). An FAQ block on the page is still useful for visitors, but expandable questions will not appear in Google results.

Do you need an llms.txt file?

llms.txt is a file proposed by Jeremy Howard in September 2024. It sits at the site root (/llms.txt) and uses Markdown to briefly describe the website and its key pages so that AI agents can use the site more easily. It is a proposal, not an adopted standard.

  • None of the search engines mentioned here has confirmed that it uses llms.txt for answers or citations.
  • Google states plainly that you do not need to create new machine-readable files or AI text files to appear in AI Overviews and AI Mode.
  • The file does no harm and is quick to create for a small website, so you can have one, but do not expect a noticeable effect.

Does speed matter?

Speed matters first of all to your visitors and to Google Search. Assess the Core Web Vitals measured on real visitors’ devices: a page is in good shape when, for 75% of visits, LCP is within 2.5 s, INP within 200 ms and CLS within 0.1. Check the mobile version, as Google indexes the mobile version of pages.

Core Web Vitals thresholds: LCP up to 2.5 s, INP up to 200 ms, CLS up to 0.1
Core Web Vitals thresholds according to web.dev.

How can we help?

Saitai.lt checks your website against this list and fixes the obstacles it finds: robots.txt, indexing in Google and Bing, structured data, page structure and consistency of your business details.

If you want AI to answer visitors on your own website, see AI search for websites; prices for all AI solutions are on the AI solutions cost page.

Written by Saitai.lt team. Published , updated .

Sources

  1. Google Search Central: AI features and your website (external website)
  2. Google Search Central: Google common crawlers (Google-Extended) (external website)
  3. OpenAI: Overview of OpenAI Crawlers (external website)
  4. Perplexity: Perplexity Crawlers (external website)
  5. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (external website)
  6. Bing Webmaster Tools: Which crawlers does Bing use? (external website)
  7. Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools (February 2026) (external website)
  8. Google Search Central: Understand JavaScript SEO basics (external website)
  9. Google Search Central: Creating helpful, reliable, people-first content (external website)
  10. Google Search Central: Add a byline date to Google Search results (external website)
  11. Google Search Central: Organization structured data (external website)
  12. Google Search Central: Article structured data (external website)
  13. Google Search Central: documentation updates (FAQ rich results) (external website)
  14. The llms.txt proposal (Jeremy Howard, 2024) (external website)
  15. web.dev: Web Vitals (external website)
  16. Google Search Central: Mobile-first indexing best practices (external website)

FAQ

Frequently asked questions

Can I stop AI models training on my content but stay in AI search?

Yes. OpenAI and Anthropic separate their search crawlers from their training crawlers, so you can block GPTBot and ClaudeBot while allowing OAI-SearchBot and Claude-SearchBot. Perplexity states that PerplexityBot is not used for training. Blocking Google-Extended does not affect Google Search, but it limits the use of your content for Gemini models and grounding their answers, so make that choice deliberately.

How long before changes show up in AI answers?

No search engine promises a fixed timeframe. Changes only reach AI answers once a crawler revisits the page. In Google Search Console you can request reindexing through the URL Inspection tool, and you can notify Bing and other engines that support IndexNow about updates through IndexNow. Even so, nobody guarantees indexing or citation.

How can I tell whether AI search is citing my website?

The AI Performance report in Bing Webmaster Tools shows how often your pages are cited in Microsoft Copilot and Bing AI summaries, and for which queries. In Google Search Console, traffic from AI Overviews and AI Mode is included in the Performance report under the Web search type, but is not shown separately. For ChatGPT and Perplexity, you are left with manual checks and referral traffic in your analytics.

Is this checklist enough to get my site cited by AI?

Not always. The checklist ensures your site can be found, crawled and understood, but whether it gets cited also depends on how useful and specific your content is and on mentions of your business on other websites. Google also stresses that indexing and serving are not guaranteed even when every requirement is met. We cover content and mentions in a separate article on ChatGPT answers.

Services

Related services

Blog

More articles on this topic

All articles

Enquiry

Have a question about your project?

Tell us what you want to achieve – we will reply and suggest a solution. The consultation is free.

+370 670 99265Mon–Fri 9:00–18:00

info@saitai.ltWe reply within 1 working day

Alaušo g. 1C, LT-08424 VilniusMeetings by prior arrangement by phone only

Enter a number

— required field. We use your data only to reply to your request.