Should I block GPTBot in robots.txt? (2026)

Clerion Research Team17 October 202613 min read

Should I block GPTBot? For most startups, no: GPTBot only collects pages for OpenAI's model training, while ChatGPT search uses OAI-SearchBot, so you can block training and still be cited. Block it if your content is the product, and never block every OpenAI bot at once.

Near-black gradient with blue light rising from below, banner reading Blocking GPTBot

Should I block GPTBot? For most startups in 2026, no, but blocking it is less risky than people think. GPTBot only collects pages for OpenAI's model training. ChatGPT search uses a separate crawler, OAI-SearchBot, so you can block training and still be cited. The mistake is blocking every OpenAI bot at once.

That split is really the whole answer, and most of the pages that rank for this question still miss it. Some are from 2023 and only know about GPTBot. Others say a GPTBot block removes you from ChatGPT answers. OpenAI's own documentation says the opposite. Below, we go bot by bot, give you a robots.txt you can paste, and cover what's changed since September.

What is GPTBot, and what does blocking it actually do?

GPTBot is OpenAI's training crawler. According to OpenAI's crawler documentation, it collects content that may be used to train its generative AI foundation models. Disallowing it tells OpenAI not to use your pages in future training. It does not remove your site from ChatGPT search.

OpenAI runs four named agents, and each setting is independent. The documentation says so in plain words: you can allow OAI-SearchBot to appear in search results while disallowing GPTBot to opt out of training. If you allow both, OpenAI may reuse one crawl for both jobs to avoid hitting your server twice.

So the question is narrower than it sounds. You're deciding whether your public pages get to shape future models, not whether ChatGPT can recommend you today. That second call belongs to a different user agent.

There's a fair case for leaving GPTBot alone, too. If your name and what you do are in a model's training data, it can mention you even when it isn't searching the web. For a small startup, being known by default is worth a lot. For a publisher whose articles are the product, it's exactly what gets given away.

Does blocking GPTBot stop ChatGPT from citing my site?

No. ChatGPT's search answers come from OAI-SearchBot, not GPTBot. OpenAI's documentation states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that changes take about 24 hours to apply. Block GPTBot alone and your search visibility stays where it was.

This is where plenty of advice goes wrong. One June 2026 guide from a bot-security vendor warns that a site-wide GPTBot disallow can also remove you from ChatGPT search citations. OpenAI's own page describes the bots as independent. When the two disagree, go with the source that runs the bot.

The third agent matters more than most people think. ChatGPT-User fetches a page when a person asks ChatGPT something that needs it, or when a Custom GPT takes an action. OpenAI says robots.txt rules may not apply to it, because a human started the request. And it isn't used to decide what shows up in search.

A Search Engine Journal report from 14 August 2026 put numbers on that. Citing TollBit's State of the Bots data for the first half of 2026, it found about 15% of identified AI page-fetchers on European sites reached URLs those sites had disallowed. ChatGPT-User reached disallowed pages on nearly half of the European sites that listed it.

Which AI bots should I allow or block in robots.txt?

Allow the search and user-action bots if you want AI assistants to cite and recommend you. Treat training bots as a separate, optional decision. Every major lab now splits its crawlers by job, so one table covers most of what hits a startup's site in 2026.

Company User agent What it does Respects robots.txt? Default for a startup
OpenAI GPTBot Collects pages for model training Yes Your call
OpenAI OAI-SearchBot Indexes pages for ChatGPT search answers Yes, about 24h to apply Allow
OpenAI ChatGPT-User Fetches a page when a user asks May not apply Allow
Anthropic ClaudeBot Collects pages for model training Yes Your call
Anthropic Claude-SearchBot Indexes pages for Claude's search answers Yes Allow
Anthropic Claude-User Fetches a page when a Claude user asks Yes Allow
Google Google-Extended A token, not a crawler: controls Gemini training and grounding Yes Your call
Google Googlebot Google Search, including AI Overviews Yes Allow
Perplexity PerplexityBot Indexes pages for Perplexity search Yes Allow
Perplexity Perplexity-User Fetches a page when a user asks Generally no Allow

The sources: OpenAI, Anthropic, Google's crawler list and Perplexity, each read on 17 October 2026. Anthropic says all three of its bots honour robots.txt and the non-standard Crawl-delay rule. Perplexity's position on Perplexity-User comes from the Search Engine Journal piece above.

Google is where people trip up. Its documentation, last updated on 15 October 2026, says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It also does not mention AI Overviews. Those are built from the normal Search index that Googlebot crawls. If you want out of AI Overviews, PPC Land's explainer notes you need snippet controls such as nosnippet or noindex, which also cost you regular search visibility. Plenty of guides still claim that blocking Google-Extended keeps you out of AI Overviews. It doesn't.

How much traffic do AI crawlers send back for what they take?

Not much, and that's the best case anyone has for blocking training bots. Cloudflare measures it as a crawl-to-refer ratio: pages crawled for each visitor sent back. In July 2026, Cloudflare Radar data put OpenAI at 251 crawls per referral, Anthropic at 1,917 and Google at 4.7.

Those July 2026 figures come from Cloudflare Radar's crawl-to-refer endpoint, as compiled by eCorpIT in August 2026. A year earlier the gap was far wider. Cloudflare's own August 2025 analysis averaged Anthropic at about 147,755 crawls per referral and OpenAI at about 1,438 for January to July 2025. Both have come down a long way since then. Google is still in a different league.

Cloudflare adds a caveat worth repeating: native apps such as ChatGPT and Claude often send no referrer header, so the ratios overstate the imbalance by an amount nobody can measure. Some of that "missing" traffic does reach you. It just lands in your analytics as direct, where nobody looks for it.

Why they crawl matters as well. Cloudflare's 2025 analysis found about 80% of AI crawling over the previous twelve months was for training, 18% for search and 2% for user actions. Radar's newer classification, as reported by TechnologyChecker on 1 October 2026, put training at 47.62% of AI crawler requests in Q3 2026, search at 8.26% and user actions at 3.66%. Cloudflare changed how it labels bots in between, so don't compare the two years head to head. Either way, most AI crawling feeds training, not answers.

Should I block GPTBot if my content is the product?

Block GPTBot when your content is the product: paid research, a dataset, a course, gated docs or original writing you sell. Leave it open when your content exists to explain your product. For most AI startups, the site is marketing, and being in the training data is free distribution.

Here's a quick way to think it through. Say you run a small AI meeting-notes tool. Your site is a homepage, a pricing page, a dozen docs pages and a blog. None of it is for sale. Every one of those pages exists so that someone, human or model, understands what you do. Blocking GPTBot saves you a bit of bandwidth, and it costs you the chance that ChatGPT knows your name before it even searches.

Now say you run a newsletter with a paid archive of market research. That archive is what people pay you for. Let it into a training set and a model can summarise it for free. Block GPTBot and ClaudeBot, disallow the archive paths for everyone, and keep the search bots open on your public marketing pages only.

Most startups sit somewhere in between. The middle path is path-level rules: allow the training bots on public marketing pages, disallow them on docs, changelogs or anything you would not want paraphrased without a link. Robots.txt lets you do this per user agent, so you don't have to make an all-or-nothing call.

What should my robots.txt look like in 2026?

Name each bot explicitly, allow the search and user agents, and make the training decision on its own line. A wildcard group does not cover a bot that has its own named group, so list every agent you care about. Here is a starting point for a startup that wants citations and is undecided on training.

# AI search and user-action agents: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI training: allow marketing pages, keep private paths out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
Disallow: /app/
Disallow: /account/

# Everyone else
User-agent: *
Disallow: /app/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

To opt out of training completely, change the training group to Disallow: /. Keep the search group exactly as it is. Then open your live robots.txt in a browser and check it's the file you think it is. Hosting platforms and CDNs can rewrite it, which brings us to the next problem.

Does robots.txt actually stop AI crawlers?

Only the ones that choose to obey it. Robots.txt is a request. It was never a lock. Named crawlers from OpenAI, Anthropic, Google and Perplexity say they follow it, but user-triggered fetchers often do not, and tools sitting in front of your site can change the file without you noticing.

The clearest recent evidence is HasData's AI Crawler Block Index, reported by PPC Land on 8 October 2026. Of 592 sites in its test set that disallowed GPTBot in robots.txt, 234 (39.5%) still served a live page to a request using GPTBot's user agent. Those requests did not come from OpenAI's verified IPs, so they show what the server enforces, not what OpenAI does.

The same study caught something quieter. Cloudflare's 15 September 2026 update retired its managed robots.txt in favour of a new Bot Preference Sync setting. HasData found 118 sites whose GPTBot ban, ai-train=no content signal and EU rights reservation had all disappeared from robots.txt by 16 September. In a live test of 21 affected sites, 16 still blocked or challenged GPTBot at the network level, but the public signal was gone. If you're on Cloudflare, check your live file today. One caveat: HasData sells scraping services, so weigh its framing with that in mind.

If you need a hard block, robots.txt won't get you there on its own. OpenAI publishes IP ranges for each of its bots, and a firewall rule can verify them. Anthropic does not publish its IP ranges and says blocking by IP is not a reliable opt-out.

What else changed for AI crawlers in autumn 2026?

Two things worth knowing this month. Meta's new commercial terms, effective 30 October 2026, let it crawl the sites of businesses that use its commercial products and train on what it finds. And Google updated its crawler documentation on 15 October 2026, restating that Google-Extended has no effect on search ranking.

The Meta change is the bigger deal. PPC Land reported on 16 October 2026 that Meta's revised terms authorise automated crawling of domains "associated with" a business's use of Meta products, and use of that content to train Meta's products. The terms do not name a user agent and do not mention robots.txt. Meta's spokesperson said site owners can still manage access through robots.txt. If you run Meta ads, read Section 1 of the new terms before 30 October and decide whether your robots.txt should name Meta's crawlers too.

None of this changes the core answer. What it does change is how often you should look. AI crawler policy now moves monthly, and a robots.txt you wrote in 2024 is almost certainly describing bots that aren't the ones visiting you now.

Do this tonight

  1. Open yourdomain.com/robots.txt in a browser. Write down every AI user agent it names, and note any that are missing from the table above.
  2. Decide on training once, for your whole site, using the content-is-the-product test. Write the decision in a comment at the top of the file.
  3. Make sure OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed on your public pages. Blocking any of them takes you out of that assistant's search answers.
  4. If you use Cloudflare or another CDN, compare the live file with the one in your repo. They may differ since September.
  5. Ask ChatGPT, Claude and Perplexity the question your product answers. Note whether you appear and which page they cite. Repeat in 30 days.

If you want to see which assistants actually send people after you change the file, Clerion reports ChatGPT, Perplexity, Claude, Gemini and Copilot as their own channel. Its free AI crawler checker shows what AI crawlers can see on your site today.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only gathers pages for OpenAI model training. ChatGPT search answers rely on OAI-SearchBot, which you control separately in robots.txt. Block GPTBot alone and you can still be cited in ChatGPT search.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects pages for training. OAI-SearchBot indexes pages for ChatGPT search results. ChatGPT-User fetches a page when a person asks ChatGPT something, and OpenAI says robots.txt may not apply to it because a user started the request.

Does blocking Google-Extended keep me out of AI Overviews?

No. Google-Extended controls whether content is used for Gemini training and grounding. Google says it does not affect Search inclusion or ranking. AI Overviews come from the normal Search index, so only snippet controls or noindex keep a page out.

How long does a robots.txt change take to reach ChatGPT?

OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust to a change for OAI-SearchBot.

Does robots.txt actually stop AI crawlers?

Only crawlers that choose to follow it. OpenAI, Anthropic, Google and Perplexity say their named crawlers do, but user-triggered fetchers like ChatGPT-User and Perplexity-User may not. For a hard block you need firewall rules, using published IP ranges where they exist.

Should an AI startup block ClaudeBot too?

Treat ClaudeBot like GPTBot: it collects pages for Anthropic model training. Blocking it is a training decision. Keep Claude-SearchBot and Claude-User allowed if you want Claude to find and cite your pages.

Sources

should i block gptbotgptbotrobots.txtai crawlersoai-searchbotai visibility
17 October 2026 · 13 min read

Clerion is website intelligence for AI founders.

Web analytics, AI traffic, conversions, errors and speed from one script tag.

Start a 14-day trialSee pricing

Keep reading

All writing →

Written from real site data by the team building Clerion. Find out whether ChatGPT, Claude and Perplexity can read your site. Check your site.