Fake AI Crawlers Are Scanning Servers for Secrets
Security firm GreyNoise has documented 824 IP addresses impersonating AI crawlers from OpenAI, Anthropic, Google, Perplexity, and others while scanning websites for exposed .env files, cloud credentials, and git config. The fakes never request robots.txt, which real crawlers always do. Site owners should verify crawler IPs, alert on credential paths, and rotate any key that was ever web-reachable.
On this page
What GreyNoise found
GreyNoise published its analysis on August 28: automated scanners are forging the crawler user agents of AI companies, including Anthropic, OpenAI, Google, and Perplexity, and pointing them at websites [1]. Between July 28 and August 23, six forged AI crawler names arrived from a single HTTP client fingerprint, joined by two of Amazon's crawler names in even greater volume [1].
The scale is not trivial. The six-name traffic came from 824 separate IP addresses spread across 795 different networks, and the same fingerprint carried more than 1,500 distinct user agent strings over 90 days, peaking on August 23 [1]. One detail stands out: 263,849 sessions claimed to be "Google-Extended", a crawler name Google says does not even have its own HTTP user agent string, so every one of those sessions was fake by definition [1].
What the fakes want, and the robots.txt tell
The scanners hunt for secrets: environment files such as .env, cloud access keys like /.aws/credentials, private keys, password stores, and /.git/config. Across the fingerprint's traffic, requests to those credential paths ran into the millions [1].
There is a reliable tell. Genuine crawlers request /robots.txt before crawling, and the forged names in this campaign never requested it even once. Anthropic's real crawler, by contrast, requested robots.txt more than any other path, about 12 percent of its traffic, and never touched a credential file [1]. The fake "ClaudeBot" string matched the genuine article character for character, but none of the 824 addresses appeared in any published crawler range from the five companies [1]. Coverage from Help Net Security and other outlets amplified the warning through the weekend [2][3].
Why impersonating AI crawlers works
AI companies publish their crawler names precisely so site owners can identify and allowlist them, and user agent strings are client-supplied headers with no built-in verification. A forged name inherits whatever trust a site grants to the real one [1]. As AI crawler traffic has grown, more sites have opened doors to it for indexing, attribution, or traffic reasons, which gives attackers a crowd to hide in [1].
GreyNoise notes that it found no evidence files were actually returned or organizations breached in this campaign [1]. The risk is the infrastructure lying around waiting to be found: an exposed .env file turns one scan into full access to your cloud accounts and API bills.
How to protect your server
GreyNoise's guidance is concrete. Never treat a user agent as identity: verify connecting IPs against each vendor's published address list. Alert on requests to /.env, /.aws/credentials, or /.git/config. Treat a crawler that skips robots.txt as suspicious, judged across days rather than a single visit [1].
Housekeeping matters just as much: keep .env, .git, and credential files out of web roots entirely, and rotate any cloud key that was ever web-reachable [1]. GreyNoise also cautions against blocking Anthropic's real indicators, since the legitimate crawler is not the problem [1]. If your site allowlists AI crawlers today, the audit is worth an hour this week.