How to make your website readable by AI crawlers in 2026 (GPTBot, ClaudeBot, PerplexityBot)

Updated September 2026 · 9 min read

A website is readable by AI crawlers when the bots behind ChatGPT, Claude, Perplexity and Gemini can fetch its pages and find the words in the HTML they receive. That comes down to seven things: an explicit Allow for each crawler you want in robots.txt, no CDN firewall rule blocking them, page text that is present without JavaScript, a title and meta description on every page, one main heading, a sitemap, and structured data where it fits. Everything else people sell under the names AEO and GEO, from content chunking to special AI files, either does nothing or applies only to a narrow case. Test where you stand with the free AI crawler checker, then work through the list.

Who is reading: the crawlers and what each one does

Every assistant runs more than one bot, and they do different jobs. Blocking one does not block the others.

Google's AI Overviews and AI Mode are built on the normal search index, so they are governed by Googlebot and normal indexing rules. Google-Extended controls whether Gemini trains on and grounds with your pages. Blocking it does not remove you from search or AI Overviews.

Fix 1: name the crawlers you want in robots.txt

Wildcard rules apply to any bot not named. That is fine if your wildcard is Allow: /. It is a problem when a plugin or template added Disallow rules years ago. Name the bots you want and allow them explicitly, so a later wildcard change cannot lock them out by accident.

User-agent: *
Allow: /
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

If you want to be cited but not trained on, leave GPTBot, ClaudeBot and Google-Extended out of the Allow list and add a Disallow: / under each. The search and user-fetch bots stay allowed. Our own robots.txt at getclerion.com allows all of them, because we want to be quotable, and we say so in a comment in the file. Decide on purpose either way.

Fix 2: check the firewall, not just the file

robots.txt is a request. A firewall is a wall. Cloudflare, Vercel, Akamai and others offer AI bot blocking as a toggle, sometimes on by default on new plans, and a WordPress security plugin can rate-limit or challenge unknown user agents. The symptom is a robots.txt that says Allow while the crawler gets a 403 or a challenge page.

Test it from a terminal with a crawler's user agent:

curl -A "PerplexityBot" -I https://yoursite.com/

You want a 200. If you get anything else, find the bot setting at your CDN or host and exempt the crawlers you named in fix 1.

Fix 3: put the words in the HTML

This is the fix that matters most and gets skipped most. Vercel and MERJ analysed crawler traffic across sites on Vercel's network and found no evidence that GPTBot executes JavaScript across more than 500 million fetches. ClaudeBot downloaded JavaScript files in about 24 percent of requests and PerplexityBot and others behaved the same way: they read the script as text and never run it. Googlebot renders JavaScript. The assistant crawlers do not.

So a page built as a client-side application, where the server sends an empty container and the browser fills it in, reads as empty to every assistant except Google's. The fix depends on your stack:

• Next.js, Nuxt, SvelteKit, Astro, Remix: use server rendering or static generation for public pages. These frameworks do it by default; the problem is usually a page that was switched to client-only rendering for convenience.
• Create React App, Vite single-page apps, Angular without Universal: add prerendering at build time for public pages, or a rendering service in front of the site.
• Webflow, Squarespace, WordPress, Shopify: the platform renders HTML on the server. Check any embedded widgets or app blocks that load content by script, such as reviews and pricing tables. That content is invisible.

Check any page with View Source. Search for a sentence from the body. If it is not in the source, the crawler never sees it.

Fix 4: title, meta description and one heading

Crawlers and the models behind them use the title tag and meta description to decide what a page is about before they read further, and the main heading to anchor the page. Every public page needs all three, written for a person: what the page is, in plain words. A title of "Home" or a heading that is a logo image tells an assistant nothing.

Fix 5: a sitemap that lists what you want found

A sitemap tells crawlers every page you want them to know about, including the ones no menu links to. Reference it from robots.txt. Keep it to pages you want cited: your comparison pages, pricing, explainers and product pages. Leave out tag archives and search result pages.

Fix 6: structured data where it describes something real

Structured data is not required for any assistant, and Google's guidance says there is no special schema for AI. It is still worth adding where it describes a real thing: an Organization with your legal name and alternate names, a SoftwareApplication or Product with price, an Article with author and date, a FAQPage for a real FAQ. It removes guesswork about facts you would rather they get right. Our own Organization schema spells out that Clerion is not Celerion or Clarion, because assistants kept confusing them.

Fix 7: content that is worth reading

None of the above helps if the page has nothing to say that a hundred other pages do not. Google's 2026 guidance on generative AI features puts this first: unique, first-hand, non-commodity content that answers the question completely on one page. The pages assistants send people to are comparison pages, pricing pages and plain explainers, because those answer the question the person asked. How to get your startup recommended by ChatGPT, Perplexity and Google AI goes through what that means in practice.

What you can skip

Google published a list in 2026 of things people are told to do for AI search that its systems do not use. Other assistants have not published equivalent lists, but the same logic applies to them.

• Chunking content into short fragments. Models handle long pages. Write for the reader.
• Rewriting text into question-and-answer form everywhere. A real FAQ is useful. Turning every paragraph into a question is not.
• Chasing every keyword variant. Models understand synonyms. One good page beats twelve thin ones, and Google treats scaled variants as spam.
• Buying or faking mentions. Assistants read the same web search does; spam systems filter the same way.
• llms.txt for Google. Google ignores it. Some coding agents and tools read it, and it costs an hour, so it is not harmful. See what is llms.txt and do you need one.

How to know it worked

Two signals, one leading and one lagging.

The leading signal is crawls. In your server logs or CDN bot analytics, the assistant crawlers should start fetching your pages within days of the fixes, and the pages they fetch most are the ones they consider worth reading. Browser-side analytics will never show these, because crawlers run no script.

The lagging signal is referrals: people clicking from an answer to your site. ChatGPT tags its links with utm_source=chatgpt.com; Perplexity, Claude and Gemini pass their domains as the referrer when they pass anything. How to track ChatGPT traffic to your website covers the setup in Google Analytics 4 and in tools that group the channel automatically. Clerion, which we build, reports AI assistants as their own channel and reads it alongside the rest of your traffic, so you can ask which pages assistants send people to and whether those visitors sign up.

Frequently asked questions

Should I block AI crawlers to protect my content?

That is your call, but separate the jobs. Blocking GPTBot, ClaudeBot and Google-Extended opts you out of training. Blocking OAI-SearchBot, Claude-SearchBot and PerplexityBot removes you from answers and the visitors they send. Most businesses want the second group allowed.

Does Google-Extended affect AI Overviews?

No. AI Overviews and AI Mode use the normal Google index. Google-Extended controls Gemini training and grounding only. To stay out of AI Overviews you would use the same snippet controls that apply to normal search results.

My site is a React app. Is it invisible to ChatGPT?

If the server sends an empty container and the browser builds the page, yes, to every assistant except Google. Move public pages to server rendering or prerender them at build time. View Source shows you what the crawler receives.

Is there one tool that checks all of this?

The AI crawler checker covers robots.txt, JavaScript-free readability, title and meta description, llms.txt, structured data, sitemap and heading in one run, free, with no signup. It checks the homepage you enter; run your most important pages too.