RobotsObservatory
You probably found this URL in an access log. Here is exactly what this bot did, and how to stop it.
RobotsObservatory is a research crawler that measures how the web
declares its policy toward AI agents. It reads robots.txt
and three well-known policy files from a fixed list of about 660 sites, once a
week, and publishes the aggregate result as an open dataset.
It does not read your pages. It requests only files that exist to be read by machines. It never touches anything behind a login, never submits a form, never solves a CAPTCHA, and never works around a paywall.
How to stop it
Put this in your robots.txt:
User-agent: RobotsObservatory
Disallow: /We read robots.txt once — that file is exempt from
robots.txt rules under
RFC 9309, which is
the only way any crawler can learn it has been told to stop — and then request
nothing else from your host, this week or ever.
No custom directive to learn, nothing to email, and nothing for us to approve.
What it requested
Per host, per weekly run, exactly five requests:
| Path | Count | Why |
|---|---|---|
/robots.txt | 2 | once per requester identity, to see whether declared policy varies by who asks |
/llms.txt | 1 | machine-readable usage terms |
/ai.txt | 1 | training opt-out convention |
/.well-known/tdmrep.json | 1 | TDM Reservation Protocol |
Five requests per host per week. 260 per year. Spread over about nine seconds, with 1.5 seconds between requests to any one host, six hosts in flight at a time. That is less traffic than one person loading one page.
The two user-agents
One declares itself. The other is a browser-shaped control, used to detect sites that serve different policy to bots than to browsers — it still carries full contact details, so no request this project makes is anonymous:
RobotsObservatory/1.0 (+https://florianhoehn.github.io/robots-observatory/bot/)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/141.0.0.0 Safari/537.36
+ From: and X-Observatory-Contact: headers pointing hereWe never impersonate another operator. No GPTBot, no
ClaudeBot, no CCBot, no third party's token, ever.
What it stores
- Status codes, response times, content hashes, and a fixed allowlist of response headers.
- The full text of your
robots.txt— that file is the subject of the study. Set-Cookieis recorded as present or absent, never by value.- Page bodies are never retained, because pages are never requested.
Removal from the published dataset
Blocking us in robots.txt stops the crawling. If you also want a
host's rows removed from the published dataset — which names roughly 660
sites and will stay online indefinitely —
open an
issue and it will be removed. You will not be asked to justify it.
Who and why
A personal research project. Every argument about AI agents and the web runs on anecdote, because there is no public baseline for what sites actually declare. This builds one, in the open, under CC0.
The collector is two readable Python files. The claim that it only ever requests the four paths above is checkable from the source, not just asserted — read it.