AI crawlers & robots.txt
Also known as: GPTBot, ClaudeBot, Google-Extended
The bots that AI companies use to read the web — and the robots.txt file where you decide which ones may read your site.
Last revised 2026-09-07. Also available as markdown — request this URL with Accept: text/markdown.
In plain English
AI companies send out crawlers (GPTBot, ClaudeBot, PerplexityBot and others) to read web pages — some to train models, some to answer questions and cite sources. Your robots.txt file is where you tell them, politely, who's allowed in.
It's a real decision: blocking training crawlers keeps your content out of models but won't affect Google ranking; allowing answer engines can win you citations and traffic. Leaving it unspecified just means “no stated preference.”
How to fix / set it up
- Decide per crawler: training vs answer engines vs user-triggered fetches.
- Generate an explicit policy with our AI robots.txt Generator.
- Merge those groups into your existing robots.txt at the site root.
AI robots.txt Generator →AI Readiness Checker →
The technical detail
robots.txt uses User-agent: groups with Allow/Disallow. Note that Google-Extended and Applebot-Extended are opt-out tokens (they gate AI training, not normal crawling), and robots.txt is advisory — only well-behaved bots obey it.
FAQ
Does blocking GPTBot hurt my Google ranking?
No — GPTBot is OpenAI's crawler and has nothing to do with Googlebot or Google rankings.
Does robots.txt actually stop AI crawlers?
Only well-behaved ones. For hard enforcement, block by user-agent or verified IP at your server, CDN or WAF.