What a crawler name is worth
A user-agent string costs nothing to type and nothing verifies it. So this asked sixteen identities to fetch the same eleven pages, three times each, and recorded what came back. Sites answer a name in four different ways, three of which any name absent from a list walks straight through. One site catches the lie. And on one of them the crawler is handed 1,640 words where a browser gets one.
The claim under test
The prompt for this was an observation going around: that spoofing is worth doing again, and that inventing a plausible AI-company user-agent gets you served like a first-class citizen. Two strings were named specifically, OpenAI File Downloader and XaiImageApiFetch/1.0. Neither is a documented crawler token. Both are in the identity list below for exactly that reason, and both turn out to work.
What the survey sends
Sixteen identities, grouped by what they claim. Three are controls, and they carry the weight of the whole table.
The row labelled ua-survey (ours) is the honest one. It claims nothing it cannot back and points at this page. Read it against googlebot further down, because the gap between those two rows is the argument in miniature.
The control is what makes the table readable
A crawler row returning 403 has two explanations that look identical from one sample. The origin refuses that name, or the origin refuses this instrument and would refuse anything wearing any name. Chrome, curl, and our own string answer the second question, so a refusal below is about the name only when a control got through.
That check disqualifies four of the eleven targets, which is the point of running it. On medium.com and quora.com every identity gets 403 including Chrome, so nothing on those hosts is evidence about user-agent policy. Reporting them as sites that block AI crawlers would be reporting our own exclusion as their policy.
ok a readable document · blocked refused · shell a 200 carrying under 50 words · word counts are the median of three trials with scripts and styles stripped
Answer one: a list of names, and it fails open
The New York Times refuses five identities and serves everybody else. GPTBot, ClaudeBot, Claude-User and PerplexityBot each get 403 and the same 135-word error page, identically across all three trials, and Amazonbot gets a 403 carrying a different 21-word body, which suggests two rules rather than one list. Chrome gets the paper. So does curl.
So do both invented strings. OpenAI File Downloader was served 1,255 words, the same count Chrome received, and XaiImageApiFetch/1.0 was served 1,438. The claim that started this checks out precisely: two made-up names cleared a wall that four real ones could not, and one of them sounds more like OpenAI than GPTBot does.
A denylist has to enumerate the thing it stops, so every identity it has never heard of sits on the allowed side by default. The population it actually filters is the crawlers polite enough to announce themselves with a documented token. Declaring yourself is what gets you blocked.
Answer two: a price
The Atlantic does something more interesting than refusing. Applebot, ClaudeBot, Claude-User and PerplexityBot each receive 402 Payment Required and a seven-word body reading {"message":"Please contact the site owner for access."}, while the other twelve identities receive the full 1,344-word homepage. 402 is the status code HTTP reserved in 1997 and left undefined for twenty-odd years, and it is now carrying pay-per-crawl.
Read what it is aimed at, though. GPTBot pays nothing and gets the homepage, and so do both fabricated names. The bill is addressed to a string, so this is the denylist above with a different status code, and it inherits the same hole. Charging for access is a real change in posture and it is orthogonal to being able to tell who is asking.
Answer three: reputation, and it fails closed
LinkedIn inverts the denylist. An earlier run against a public profile, taken before this survey had made many requests, put three clean bands on the same URL: about 11,800 words to Chrome, about 10,400 to ClaudeBot, and about 7,900 to Claude-User, with Contact Info present for the browser and absent for every crawler. Both invented strings got 999, LinkedIn's house refusal code.
An unknown name is refused instead of admitted, which stops the trick that works on the Times. It also refuses a real, well-behaved, newly launched crawler on its first request and every request after, because being unknown is the disqualifying property rather than behaving badly.
The survey broke this row by running it. By the final pass LinkedIn was answering 999 to fifteen of sixteen identities including Chrome, so the control gate marks the target unmeasurable and the bands above are no longer reproducible from this address. The numbers are real and were taken before the block. An instrument that changes its subject by measuring it has to say so, and a survey that keeps sampling a defended origin is training that origin to refuse it. The same thing happened to stackoverflow.com partway through, which slid from 403 into 429.
Answer four: verification, and it catches the lie
Wikimedia is the only site in the sample doing something a string cannot defeat. It served the full 8,243-word article to Chrome, to curl, to our own invented survey string, to GPTBot, to Claude-User, to PerplexityBot, and to both fabricated names. It refused six: googlebot, bingbot, Applebot, DuckDuckBot, facebookexternalhit and ClaudeBot, each with a 9 or 10 word body reading Unauthorized request. Please contact bot-traffic@wikimedia.org to request unblocking.
Every one of those six publishes the addresses its crawler uses, or a reverse DNS procedure, or both. Claiming to be nobody is fine. A nonsense token invented on the spot is fine. Claiming a checkable identity from an address that cannot back it is refused. Three AI crawlers read the article in full during the same run, so what Wikimedia turns away is impersonation rather than crawling. The honest versions of those six are unaffected, because their addresses match what they say.
One caveat kept because it spoils the tidy version: GPTBot passes, and OpenAI publishes crawler addresses too. So the check is applied unevenly rather than universally, and the honest claim is narrower than the pattern invites. What held across every trial is the contrast between claiming nothing and claiming Googlebot.
The direction nobody mentions, where the crawler gets more
Reddit is the strongest result in the survey and it points the opposite way from a paywall. A browser asking for /r/programming is served one word. facebookexternalhit, the Facebook link-preview fetcher, is served 1,640. Same URL, same address, same three trials.
The mechanism is mundane once you see it, and it is worth stating because the exciting reading is wrong. Reddit renders for humans in the browser and pre-renders for clients that cannot run JavaScript, because a link preview has to work without a JS engine. So the crawler is handed server-rendered HTML while the browser is handed an empty frame and a script tag. Nobody decided to be generous to bots.
That still leaves a real and widely useful trick, and it is a better description of what spoofing buys than the paywall story. Claiming to be a link-preview bot is a way to get server-rendered HTML out of a client-rendered application. The win is a rendering path rather than a permission, which is also why it works on sites with no AI policy whatsoever.
The sites with no opinion at all
Three targets served all sixteen identities the same document to the word: GitHub at 1,135 words, Hacker News at 747, and this site at 314. That is the baseline the other rows should be read against, and it is a reminder that most of the web is still answering a request rather than interrogating it.
What makes a name checkable
Reverse DNS is the old answer and it works. Resolve the connecting address, confirm the name belongs to the crawler's operator, resolve it forward again to confirm it points back. Wikimedia's six refusals are that check failing, and it is why a fabricated token sailed past while googlebot did not.
It is also available to about a dozen companies, because it requires owning address space and publishing a procedure that site operators bother to implement. Everybody else gets a string, which is where this page started.
Web Bot Auth moves the check off the address. The crawler signs each request with an Ed25519 key under RFC 9421 HTTP Message Signatures and publishes the public half as a JWKS document. Three headers carry it: Signature-Input naming the covered components with tag="web-bot-auth", a keyid thumbprint and created and expires stamps; Signature carrying the signature; and Signature-Agent naming the directory to fetch the key from. The directory lives at /.well-known/http-message-signatures-directory.
The property that matters is the one this survey keeps demonstrating in the negative. Anyone can type GPTBot. Producing a valid signature over a request needs the private key, so the claim stops depending on which address it arrives from, and a crawler owning no network blocks can still hold an identity somebody is able to check.
The program that already runs this, and the part still waiting
None of that is a proposal. Cloudflare's Verified Bots program accepts exactly three proofs of identity, and they map onto this page's answers one for one: a Web Bot Auth signature, a published IP list paired with a stable user-agent, or reverse DNS validation. Verification carries behavioural terms too, since a verified bot has to respect robots.txt, hold a reasonable request rate, and avoid working around a site owner's stated preferences. Breaking any of them removes the service from the global allowlist. The signature route is the one that widens the field, because the other two proofs are facts about the address a request comes from and a signature is not.
Two labels sit close together here and are worth keeping apart. A valid Web Bot Auth signature classifies traffic as a signed agent, and Verified Bot status is a further step judging behaviour rather than cryptography. Cloudflare's own announcement uses itself as the example: Browser Rendering sends signed Web Bot Auth headers and does not qualify as a verified bot, while Radar's URL Scanner does. Signing proves who is asking. Verification judges whether that identity has earned an allowlist entry.
So the fix for everything above is deployed, at one of the largest reverse proxies on the web, and a site gets it by turning it on rather than by building it. Whether anything in this sample looks for it is a separate question, and the survey carries one small piece of evidence.
That row sends the pointer and signs nothing. It sets Signature-Agent to a directory that genuinely exists and omits Signature and Signature-Input entirely, so an origin actually verifying has everything it needs to reject the request. Read it against the plain ua-survey (ours) row in the matrix, which is the same user-agent carrying no header at all.
Worth stating the limit of that probe. It shows whether an origin reacts to an unsigned Web Bot Auth attempt, and it cannot show whether an origin would honour a correctly signed one, because signing needs AadharshBot's private key and that key is a Worker secret this survey cannot reach. A run from inside the Worker would answer the second question and is the obvious next experiment.
One more data point arrived by accident while this page was being written. Cloudflare's own Radar bot directory answered 403 to the unverified fetcher used to read it, and served the same page to an ordinary browser seconds later. The registry of verified bots sits behind the wall that verification exists to open.
What this site serves
The last row of the matrix is aadhar.sh, which answered all sixteen identities with the same 314-word document. That is a deliberate position rather than an absence of policy: the pages here publish a Markdown twin, an MCP endpoint and a machine catalog precisely so a crawler has a cheaper option than scraping the HTML. Serving one document to everyone is easy when there is nothing you are trying to keep from them.
It also means this page cannot claim to have tested its own defense, because it has none to test.
The crawler is a different story, and the honest version implicates us. AadharshBot signs its outbound requests under RFC 9421 and publishes an Ed25519 key at the exact well-known path Cloudflare's documentation specifies, served as application/jwk-set+json. That clears the technical bar and stops there, because the bot has never been submitted to the Verified Bots program. So the signature it has been sending all along buys it nothing at any origin that checks a registry rather than a signature, and this page argues for a thing its own crawler has half done.
What would falsify this. The denylist reading dies if the Times refuses OpenAI File Downloader on a later run, which would mean the list is maintained faster than a survey can exploit it. The verification reading dies if Wikimedia turns out to refuse googlebot from every address including Google's, which would make it an ordinary denylist that happens to name six crawlers. The rendering explanation for Reddit dies if facebookexternalhit is served more than a browser on a server-rendered page, where there is no JavaScript gap to explain it. The adoption reading dies if any origin treats the unsigned Signature-Agent row differently from the bare row beside it. All four are one re-run of bun run ua:survey away, and the survey prints its own control rows so a future reader can tell a changed policy from a changed instrument.
