Can ChatGPT, Perplexity and Google AI Read Your Website? A 10-Minute Check

October 14, 2026Guide6 min read

AI search tools can only quote what they can fetch and read. Four checks, about ten minutes in total, tell you whether your site passes: your robots.txt, how your pages render, whether a firewall blocks crawlers, and whether the pages are indexed.

Being readable does not guarantee being cited. Which sources an answer uses depends on the platform and the question, and the platforms do not publish their ranking rules. But being unreadable guarantees you are left out, and that is the part you control.

1. Check your robots.txt

Open yourdomain.com/robots.txt. Each AI service identifies its crawler with a name, and a rule that blocks that name keeps it out. The ones worth knowing:

OAI-SearchBot: the crawler behind ChatGPT search results. GPTBot is a separate OpenAI crawler used for model training.
PerplexityBot: Perplexity's crawler.
ClaudeBot and Claude-SearchBot: Anthropic's crawlers.
Google-Extended: a token that controls whether Google may use your content for its Gemini models. It does not affect whether you appear in Google Search.

If you see a blanket "Disallow: /" for all agents, or a block on the names above that you did not choose, that is your first fix. Whether you want to be used for model training is a separate decision from whether you want to appear in AI search answers, and you can allow one and block the other.

2. Check that your text is in the page

Open your page, choose View Source, and search for a sentence from your main content. If it is there, a crawler that does not run JavaScript can read it. If the source is mostly empty and the text only appears after scripts run, some crawlers will see an empty page. Sites built to render on the server avoid this.

3. Check for a firewall or bot protection

Some hosting and security services block unfamiliar bots by default. You can request a page while presenting a crawler's user agent, for example with curl, and see whether you get a normal response or an error. This only reveals rules based on the name a crawler gives, not rules based on its network address, so treat a pass as a good sign rather than proof.

4. Check that your pages are indexed

Submit your sitemap in Google Search Console and in Bing Webmaster Tools. Many AI search features draw on a conventional search index, and the sources disagree about which one each platform leans on most, so covering both is the safe choice. IndexNow lets you tell participating engines, including Bing, when a page changes.

What is not worth your time

An llms.txt file. We found no solid evidence that having one changes whether you are cited, and Google has said that Search does not use it. It is harmless, but it is not a lever.
Adding structured data in the hope of being cited. It helps machines understand who you are, but we found no solid evidence that it raises citations by itself.
Publishing a large volume of generic articles. Answers favour pages that are specific, structured and attributable.

What does help, according to the evidence we trust

A study of 21,143 citations across ChatGPT, Perplexity, Gemini and Google AI Overviews found that cited pages tend to answer the question directly, early, and under clear headings. Research from Princeton found that adding statistics, source citations and quotations raised visibility in generated answers. Freshness matters too: a large share of cited content is recent. And being mentioned on sites that AI tools already cite, such as review sites, directories and forums, is worth more than most on-page tweaks.

Sources: "From Citation Selection to Citation Absorption: A Measurement Framework for Generative Engine Optimization Across AI Search Platforms" (arXiv:2604.25707, 2026); Aggarwal et al., "GEO: Generative Engine Optimization" (Princeton and collaborators, 2024); LLMrefs, "Answer Engine Optimization: The Complete Guide for 2026".