Everyone is counting AI crawlers now. Open the log, search for GPTBot, add it
up. That number is wrong, and not by a little.
The measurement below comes from the logs of a site under our maintenance and search work: nine days, 11,569 requests. Every request claiming to come from OpenAI was checked against the IP ranges OpenAI publishes.
The result
| agent | requests | genuine | spoofed |
|---|---|---|---|
| GPTBot | 437 | 216 | 50% |
| OAI-SearchBot | 168 | 55 | 67% |
| ChatGPT-User | 114 | 9 | 92% |
| total | 719 | 280 | 61% |
Six in ten requests that say OpenAI are not OpenAI.
The proportion rises as the agent becomes more convenient to imitate. GPTBot
collects training data and is the most widely blocked, so it is the least useful
name to borrow. ChatGPT-User fires when a person asks ChatGPT for a page — it
is the traffic nobody blocks, because it arrives on a human's behalf. That is why
92% of it is somebody else.
What the spoofed ones ask for
Not content.
GET /secrets.json 404 "...compatible; ChatGPT-User/1.0..."
GET /server.key 404 "...compatible; ChatGPT-User/1.0..."
GET /service_account.json 404 "...compatible; ChatGPT-User/1.0..."
GET /.zsh_history 404 "...compatible; ChatGPT-User/1.0..."
GET /stripe.json 404 "...compatible; ChatGPT-User/1.0..."
These are credential scanners wearing a user-agent nobody filters. Of 1,314 "AI" requests in the window, 406 returned 404 — nearly all of them this shape.
Nothing was found, because nothing is there. But a configuration file left at the root once would have been collected by a scanner dressed as ChatGPT.
Why it matters
Any AI-visibility report built on user-agent strings is inflated. If a tool tells you GPTBot visited 437 times, half of that is someone else. If it tells you 114 people asked ChatGPT about your site, the real number is nine.
The difference is not cosmetic. Nine genuine ChatGPT-User requests in nine days
means "almost nobody is asking yet", which is a useful answer. A hundred and
fourteen means "we have traction", which is false and stops you working on the
thing that actually needs work.
How to check properly
A user-agent is a string anyone can type. The only evidence is the IP address.
OpenAI publishes its ranges as JSON, one file per agent:
https://openai.com/gptbot.json— 21 rangeshttps://openai.com/searchbot.json— 35https://openai.com/chatgpt-user.json— 204
The check is whether the address falls inside one of them. A request carrying the name but arriving from outside the published ranges is spoofed, however convincing the string looks.
Google publishes the same thing for Googlebot, and there is a second method there:
a reverse DNS lookup that returns a name under googlebot.com, followed by a
forward lookup on that name. If both close, it is Googlebot.
What cannot be checked
Anthropic does not publish IP ranges. The equivalent file does not exist at any of the addresses where it would be expected.
So for ClaudeBot and Claude-User there is no public way to separate real from
fake. The careful assumption is that the proportion resembles OpenAI's, and that
those numbers are inflated too.
Perplexity is the same: the name is documented, the ranges are not.
What to do about it
Do not block by user-agent. A scanner pretending to be ChatGPT-User does not
stop because you disallowed ChatGPT-User in robots.txt — it never reads the
file. The only thing you achieve is blocking the real one.
Verify before you report. If you are building an AI-visibility dashboard, IP matching is the difference between a number and an opinion.
Look at status codes, not just visits. A high proportion of 404s from an agent that should be reading articles is the simplest signal that articles are not what it came for.