Our crawler
If you found this address in your server logs, this page explains what we were doing.
How we identify ourselves
Outranked.me/1.0 (+https://outranked.me/about/crawler)
What we fetch, and why
- A listed site’s home page, once. When a business is added to the directory we read the title, description and preview image so the listing is not blank. One request.
- A verification file, when an owner asks us to. Domain verification fetches a single file at
/.well-known/that the owner placed there deliberately. - A safety re-check of a destination URL. We re-validate where a listing points, because a destination that was safe when it was submitted may not be safe later.
- Whatever somebody types into our AI access checker. That tool fetches robots.txt and the page a visitor entered, several times, each request carrying the user-agent of the crawler being tested — because the whole purpose is to observe how a server responds to each one. Capped at 20 checks per hour per visitor.
What we never do
- We do not spider. We fetch the specific URL we were given, never links from it.
- We do not execute JavaScript, submit forms or accept cookies.
- We do not fetch on a schedule to build an index of the web.
- We stop reading after a fixed number of bytes and abandon any request that takes longer than a few seconds.
- We refuse any address that resolves to a private network, re-checking on every redirect.
How to block us
Add this to your robots.txt:
User-agent: Outranked.me Disallow: /
Two honest caveats. Our listing and verification fetches are actions a person took on purpose — adding their business, or proving they own a domain — so they behave like a link preview rather than a crawl. And when the access checker requests a page as GPTBot in order to see how you answer GPTBot, it necessarily sends that user-agent rather than ours, so your robots.txt rule for us will not describe that request. If you would rather we did not fetch your site at all, write to us and we will add the domain to a blocklist.