Blocking crawlers (Cloudflare / WAF challenges)
Also known as: Bot Fight Mode, Just a moment
When your bot protection (like Cloudflare) shows automated visitors a challenge page, it can also block the search and AI crawlers you actually want.
Last revised 2026-09-07. Also available as markdown — request this URL with Accept: text/markdown.
In plain English
Anti-bot tools like Cloudflare's Bot Fight Mode protect you from bad automated traffic by showing a “checking your browser” challenge. The catch: they can't always tell a scraper from Googlebot or an AI answer engine, so they sometimes challenge the good bots too.
When that happens, those crawlers get the challenge page instead of your content — so your pages can quietly become invisible to Google and to AI assistants, even though the site looks fine to you in a browser.
How to fix / set it up
- In Cloudflare, enable “Verified bots” and turn off Bot Fight Mode for the agents you want to allow.
- Add WAF/allow rules for at least GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended and Googlebot.
- Re-test with the AI Readiness Checker to confirm the block is gone.
AI Readiness Checker →Agent Readiness Checker →
The technical detail
Challenges are often served with HTTP 200 and an interstitial (cdn-cgi/challenge-platform, “Just a moment…”), so automated readers see no real content. Allow verified bots and add WAF skip rules for the agents you want.
FAQ
How do I know if I'm blocking crawlers?
Run the AI Readiness Checker — it detects when it's served a challenge page instead of your content and flags it.
Isn't blocking bots good?
Blocking bad bots is good; the goal is to let the verified search and AI crawlers through while keeping scrapers out.