Verifying AI crawlers
Also known as: reverse DNS verification, forward-confirmed rDNS, crawler IP ranges
Proving a visitor claiming to be GPTBot really is GPTBot — because the user-agent string is a claim anyone can copy, and blocking on it wrongly costs you the crawlers you wanted.
Last revised 2026-09-07. Also available as markdown — request this URL with Accept: text/markdown.
In plain English
Any script can announce itself as Googlebot. If your blocking rules trust the name, two things happen: attackers stroll past by borrowing it, and one day a rule catches the real crawler and quietly removes you from an index.
Verification closes both holes. There are two spoof-resistant methods, and neither involves believing the name in the header.
How to fix / set it up
- Never block on user-agent alone. Treat the string as a hint that decides which verification to run.
- Implement forward-confirmed reverse DNS for the vendors that publish PTR records.
- Fetch and cache the published IP-range files for the vendors that publish them.
- Exempt your own office and admin addresses from every automatic block, at ban time and at enforcement time — an address blocked earlier cannot reach the panel where the block is lifted.
- Keep every automatic block reversible and annotated with why it happened.
Full guide: controlling AI crawlers →AI robots.txt Generator →
The technical detail
Reverse DNS with forward confirmation: look up the visitor's IP for a hostname, check that it belongs to the vendor's domain (googlebot.com, search.msn.com, crawl.amazonbot.amazon, applebot.apple.com), then resolve that hostname back and confirm it returns the same IP. Both directions are required — one alone is forgeable.
Published IP ranges: OpenAI, Perplexity, Ahrefs and others publish JSON range lists. Fetch and cache them (a day is fine), then match on address alone. This layer keeps working when a vendor introduces a crawler name you have never heard of, which is precisely the case that breaks user-agent rules.
Run both layers, because each fails where the other holds: reverse DNS needs a PTR record that not every vendor publishes, and range lists go stale between refreshes. Cover IPv6 as well as IPv4 — an IPv6-only crawler with no PTR record is a real and easily missed case.
When a verified crawler hits a trap or a blocked path, answer 410 Gone and log it: 410 makes a legitimate crawler drop the URL permanently, while 403 invites it back tomorrow. Never ban it.
FAQ
Is the user-agent ever enough?
Only for politeness — deciding what to serve. Never for enforcement, where being wrong is expensive and silent.
What if a vendor publishes neither PTR records nor ranges?
Then you cannot verify it, and blocking on its name is guesswork. Rely on robots.txt for the compliant ones and accept that the non-compliant ones need a different control.