Field Notes
August 22, 2026 · J. I. Ashley Consulting
Last week our own audit tooling flagged something strange on one of our sites: the robots.txt we had written said every AI crawler was welcome, and the robots.txt the internet actually received said the opposite. GPTBot, ClaudeBot, Google-Extended, Amazonbot, and Applebot-Extended were all disallowed. Nobody at our firm turned that on.
The culprit was our CDN. Cloudflare ships a setting called AI bots protection along with a managed robots.txt, and on some plans and setup flows it arrives enabled. When it is on, Cloudflare injects its own directives at the edge, ahead of whatever your file says, and blocks the crawlers at the network level for good measure. Your origin can say welcome all it wants. The edge answers first, and the edge wins.
A week later we audited an agency that sells AI search optimization. Same block, same bots, near-certainly the same default. If it can happen to us and to a practitioner in the field, the base rate across ordinary business websites is high. Most owners will never know, because every internal view of the site looks fine.
ChatGPT, Gemini, Claude, and Google's AI answers are where a growing share of buying questions get asked and answered. Those systems learn about your business by crawling it. Blocking GPTBot and ClaudeBot means the assistants your customers consult are working from other people's descriptions of your market, in which you may not appear at all. Blocking Google-Extended keeps you out of the material Google's models draw on. None of this shows up in your analytics as an error. You just quietly stop being in the answers.
Fetch your robots.txt the way a crawler does, not through your CMS. From any terminal:
curl https://yourdomain.com/robots.txt
Read what actually comes back. If you see a block of directives you did not write, often between comments like BEGIN and END Cloudflare Managed Content, with lines such as "User-agent: GPTBot" followed by "Disallow: /", the edge is overriding you. Cloudflare users can also check the dashboard under Security, then Bots: look for AI bots protection set to block, and a managed robots.txt toggle.
Decide your actual policy first. A reasonable one for most businesses: welcome the crawlers that put you in front of buyers, and block the bulk scrapers that only hoard content. Then make the edge agree with it. On Cloudflare, set AI bots protection to off and disable the managed robots.txt, either in the dashboard or with one API call to the bot management endpoint. Keep your own robots.txt as the single source of truth, and verify from outside again after the change.
One caution in the other direction: some sites want these blocks, and choosing them deliberately is legitimate. The failure is not the setting. The failure is the setting being on without the owner knowing, while their file claims otherwise.
What your website says and what the internet receives are not the same thing, and the gap is invisible from inside. The only audit that means anything is run from where the crawlers stand. That principle is the foundation of our AI visibility audit, and this finding is a standing exhibit for it: two websites, both owned by people who think about this professionally, both silently invisible to the machines they wanted attention from.