How to check what a crawler actually sees

Opening your site in a browser tells you what a browser sees. That is a different question from what a crawler sees, and on a modern site the two answers can be completely different.

Here is the method we use, and — more usefully — the ways it lies.

Ask as the bot, not as yourself

The whole check is one request with the right User-Agent and a word count:

check() {
  curl -sS -A "$2" --max-time 15 "$1" | python3 -c "
import sys, re
h = sys.stdin.read()
b = re.search(r'<body[^>]*>(.*)</body>', h, re.S)
if not b: print(0); raise SystemExit
t = re.sub(r'<script.*?</script>|<style.*?</style>|<noscript.*?</noscript>', '', b.group(1), flags=re.S)
print(len(re.sub(r'<[^>]+>', ' ', t).split()))"
}

for ua in \
  "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" \
  "GPTBot/1.0 (+https://openai.com/gptbot)" \
  "OAI-SearchBot/1.0" \
  "ClaudeBot/1.0" \
  "PerplexityBot/1.0" \
  "Applebot/0.1"
do
  printf "%-24s %s words\n" "${ua:0:22}" "$(check https://example.com/ "$ua")"
done

Two things make this worth running rather than assuming.

Run it on every address in your sitemap, not the homepage. A site can render its front page server-side and leave every deep page empty. Those are the pages that would have ranked.

Compare across agents. If Googlebot gets content and GPTBot gets zero, you have a rule somewhere — in robots.txt, in a CDN, in a bot-protection product — that is deciding for you which engines may read you. That decision is worth making on purpose rather than inheriting it.

Then read your own logs

The curl check tells you what a crawler would get. Your access log tells you what actually happened, and it is the only evidence in this whole exercise that nobody else can dispute.

If your requests are logged with headers, group them by user agent and by day:

const BOT = /(Googlebot|bingbot|GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|Google-Extended|Applebot|YandexBot|Amazonbot|DuckDuckBot)/i;

for (const line of readLines(logFile)) {
  const entry = parse(line);
  const hit = BOT.exec(entry.headers['user-agent'] ?? '');
  if (hit) counts[hit[1]][entry.day]++;
}

Four months of that told us more than any tool. A sample, from the week we shipped prerendering to a site that had been serving crawlers an empty body:

Two things in that data were not visible anywhere else. Four crawlers that had never appeared — PerplexityBot, Applebot, DuckDuckBot, Amazonbot — started within a week. And ChatGPT-User, the agent that fetches a page while a person is waiting for an answer in ChatGPT, went from never to 33 requests in a day.

Also count distinct pages, not just requests. A thousand hits on the homepage and a thousand hits spread over sixty pages mean opposite things.

Three ways these checks lie

This is the part worth having. Every one of these produced a false finding in our own audit before we caught it.

curl with no User-Agent is not a crawler

We reported that a client's sitemap.xml returned 403. It did — to curl with its default agent. The site's bot protection blocks unknown clients, and curl/8.5.0 is an unknown client.

The same file, asked for properly:

curl         403
Chrome       200
Googlebot    200
bingbot      200

There was no SEO problem at all. The defect was in the instrument. Never run a crawler check without -A, and when a result looks alarming, re-run it as Chrome and as Googlebot before believing it.

The site: operator, read through a scraper, is not evidence

We used site: queries fetched from a search engine to conclude that three client sites were not indexed. Two of them were.

Search engines serve degraded pages to automated clients. In one run the fetched result page for site:createli.md was filled with results from a German mobile carrier's support forum. The reported result count was noise too: site: on a domain with no matches returned "about 2,160,000 results".

What is actually reliable, in order: Search Console, which is the engine telling you directly; a domain-restricted search through a real search API; and a plain brand query, which at least confirms something is indexed. A site: count scraped from an HTML page confirms nothing.

Half the "AI crawlers" in your logs are not AI crawlers

Grouping four months of logs by user agent produced a table where PerplexityBot had requested /.env.production, /.env.bak and /.git/HEAD; Applebot had asked for /backend/.env; OAI-SearchBot for /credentials.yaml; YandexBot for /wp-config.php.bak.

Real Perplexity does not scan for environment files. These are scanners wearing a bot's name, because bot user agents are often exempted from rate limits.

Two consequences. Your crawler statistics are inflated unless you verify — by reverse DNS, or against each vendor's published IP ranges. And every one of those paths is worth checking on your own site:

for p in /.env /.env.production /.git/config /credentials.yaml /wp-config.php.bak; do
  printf "%-26s %s\n" "$p" "$(curl -sS -o /dev/null -w '%{http_code}' "https://example.com$p")"
done

Anything other than 403 or 404 is a finding. On one site every one of those returned 200, because the SPA fallback served the homepage for any unknown path — which is also, separately, an unbounded set of soft 404s.

What to conclude

The order that survives scrutiny:

1. curl per agent, across the whole sitemap. Tells you whether the content exists for a crawler at all.
2. Your own access log, grouped by agent and day. Tells you who actually came and what they took. Strongest evidence available, and it is yours.
3. Search Console. Tells you what the engine decided afterwards. Nothing else answers that question.

And one discipline underneath all three: when a result surprises you, suspect the instrument before the site. Twice in one audit, the surprising finding was our own measurement.