What this file is

The firm's own record behind the field note: the 2026-08-20 finding on jiashley.com, the Cloudflare zone settings before and after the fix, the audit tool's robots.txt checks across the firm's properties and one prospect, and the live robots.txt as served on the day this file was built.

Fields

fieldvalue
studycdn-blocking-ai-crawlers

finding_2026_08_20

fieldvalue
sitejiashley.com
recorded_inphaneia/FINDINGS.md, entry '2026-08-20 — Cloudflare was silently blocking AI crawlers on jiashley.com'
origin_robots_txt_policyAI crawlers welcome
audit_before_fixWARN, 3 AI bots blocked (recorded in the finding; the pre-fix JSON was not kept)
audit_after_fixPASS, AI bots blocked: none (jiashley.com_2026-08-20.json)
record_quote
The zone had `ai_bots_protection = block` plus Cloudflare's managed robots.txt enabled. Net effect: GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot, and Applebot-Extended were all disallowed AND blocked at the network edge. The origin robots.txt said "AI crawlers welcome"; the edge said the opposite, and the edge wins.

finding_2026_08_20.zone_settings_before

fieldvalue
ai_bots_protectionblock
is_robots_txt_managedtrue

finding_2026_08_20.crawlers_disallowed_by_the_served_robots_txt

value
GPTBot
ClaudeBot
Google-Extended
CCBot
Amazonbot
Applebot-Extended

finding_2026_08_20.fix

fieldvalue
apiPUT /zones/{id}/bot_management

finding_2026_08_20.fix.body

fieldvalue
ai_bots_protectiondisabled
is_robots_txt_managedfalse

audit_robots_checks

sitedatestatusdetail
an agency selling AI search optimization (name withheld)2026-08-21WARNpresent; AI bots blocked: ['GPTBot', 'ClaudeBot', 'Google-Extended']
anamnesis.smtry.ai2026-08-20FAILmissing or soft-404 (HTML served)
anamnesis.smtry.ai2026-08-20PASSpresent; AI bots blocked: none
jiashley.com2026-08-20FAILmissing or soft-404 (HTML served)
jiashley.com2026-08-20PASSpresent; AI bots blocked: none
jiashley.com2026-08-21PASSpresent; AI bots blocked: none
jiashley.com2026-08-25PASSpresent; AI bots blocked: none
smtry.ai2026-08-20FAILmissing or soft-404 (HTML served)
smtry.ai2026-08-20PASSpresent; AI bots blocked: none

live_capture

fieldvalue
sitejiashley.com
fetched2026-09-05
robots_txt
# jiashley.com — public consulting site; search and AI crawlers welcome.
# Bulk scrapers that hoard content without attribution are not.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://jiashley.com/sitemap.xml

live_capture.ai_crawlers_disallowed_now

value

method

fieldvalue
toolphaneia/siteaudit.py robots.txt check: fetch /robots.txt from outside, flag AI crawlers under Disallow: /
noteAudit JSON files record the tool's verdict lines, not the served file body; the served body is captured here for the live check only.

Files