Pre-run · no data yet
Robots Observatory
How does the web declare its policy toward AI agents — and does that declaration change depending on who asks?
A frozen panel of roughly 660 real websites, screened from the Tranco top
1000. Every week each host's policy surface is read —
robots.txt plus llms.txt, ai.txt and
/.well-known/tdmrep.json — under two requester identities,
and the result is published as an open dataset.
Policy only. This project reads the files the web publishes for machines to read. It never fetches a page, so it measures what sites say, not what they enforce. Those are different datasets and this is the first one.
Every argument about AI agents and the web currently runs on anecdote, because there is no public baseline for what sites actually declare. This builds one, in the open, under CC0.
The collector and its tests are written. The panel is not yet frozen and no run has happened, so there are no findings on this page yet.