Is your CDN blocking the AI crawlers you want?
Our own robots.txt said every AI crawler was welcome. The one the internet received said the opposite. The CDN was rewriting it at the edge.
J. I. Ashley Consulting
Read the pieceBlog
What we find, written down. Notes from running production AI systems and measuring what they do. Every claim here is something we tested on a real site, usually our own first. RSS
Our own robots.txt said every AI crawler was welcome. The one the internet received said the opposite. The CDN was rewriting it at the edge.
J. I. Ashley Consulting
Read the piece