Is your CDN blocking the AI crawlers you want?
Our own robots.txt said every AI crawler was welcome. The one the internet received said the opposite. The CDN was rewriting it at the edge.
Writing
Notes from running production AI systems and measuring what they do. Every claim here is something we tested on a real site, usually our own first.