Free AI checker
Can AI crawlers access your website?
A free AI access checker. Enter a website and we read its robots.txt, then request the page once as each crawler and report exactly what came back. No account, and every finding shows the evidence behind it.
We fetch your home page and robots.txt once as each agent. No account, and nothing is stored against you.
What is an AI crawler?
An AI crawler is an automated client that fetches web pages on behalf of an AI company. They fall into two groups that are easy to confuse. Training crawlers, like GPTBot, collect pages that may be used to train models. Retrieval crawlers, like OAI-SearchBot and PerplexityBot, fetch pages to answer a question someone is asking right now.
The distinction matters when you decide what to allow. Blocking a training crawler keeps your content out of a future model without affecting whether an assistant can look your site up today. Blocking a retrieval crawler does the opposite. They are separate decisions and robots.txt lets you make them separately.
How does an AI crawler checker work?
This one works by asking your website the same questions those crawlers ask. First it fetches /robots.txt and parses it properly, including the rule that a group naming a specific agent overrides the * group. Then it requests your page once for each crawler, sending that crawler’s user-agent, and once more as an ordinary browser for comparison.
Comparing those responses is the part a robots.txt reader cannot do. If a browser receives a full page and a crawler receives a challenge page, or a tenth of the text, that difference is only visible if you actually make both requests.
Which AI crawlers do we check?
What does a blocked AI crawler mean?
It means a crawler that honours robots.txt will not fetch the page. That is all it means, and it is frequently intentional — publishers, agencies and businesses with licensing concerns block training crawlers on purpose and are right to.
The checker reports which crawler, which rule and which user-agent group produced the result, then asks you to confirm it was deliberate. It will not tell you to remove the rule, because it has no way of knowing whether you meant it.
Why can robots.txt allow a crawler while the server still returns 403?
Because they are enforced in different places. robots.txt is a request the crawler chooses to honour. A 403 is your server, or more often your CDN or web application firewall, refusing the connection outright. Bot protection is usually configured once and then inherited through migrations and plan changes, so this is a common gap between what a site intends and what it does.
There is an important caveat in the other direction. Our requests come from Outranked’s infrastructure, not from the crawler’s. Googlebot and Bingbot are verifiable by reverse DNS, and the major AI crawlers publish address ranges, so a site that correctly refuses unverified claimants will refuse us while serving the genuine crawler perfectly. A refusal here is a reason to check from a verifiable address — never a reason to weaken bot protection.
Does allowing AI crawlers guarantee that ChatGPT will mention my website?
No. Technical access does not guarantee crawling, indexing, retrieval, citation, recommendation, ranking or mention. It removes an obstacle. Every other factor — whether a system fetches your page, whether it retrieves it for a particular question, and whether it quotes you in the answer — is outside what any external tool can observe.
This is worth being blunt about, because a whole category of product is currently being sold on the opposite claim. If a tool gives you an “AI visibility score”, ask what it measured. Ours gives you HTTP status codes and robots.txt rules, because those are things that exist.
Frequently asked questions
- Is the AI checker free?
- Yes, and there is no account. Checks are capped at 20 per hour per visitor so the tool stays polite to the websites being checked, since each check makes several requests to them.
- What does the AI Access Checker check?
- Four things. Whether your robots.txt allows each crawler to fetch the page, and which rule applies. Whether your server actually answers a request carrying that crawler’s user-agent. How much readable text exists in the HTML before any JavaScript runs. And what the page declares about itself: title, meta description, canonical link, noindex directives and whether structured data parses.
- Which AI crawlers are tested?
- GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, plus Googlebot and Bingbot. Google-Extended is also reported, but only as a robots.txt rule: it is a token that controls Gemini training use and is not a crawler, so there is nothing to send a request as.
- Can I check my website without signing up?
- Yes. Enter an address and the result appears on the page. We do not ask for an email and we do not store the result against you.
- Does robots.txt control AI crawlers?
- It controls the ones that choose to honour it, which the major named crawlers document that they do. robots.txt is a request rather than an enforcement mechanism: it has no technical power to stop anything, and a crawler that ignores it faces no obstacle. If you need enforcement, that is a server or edge rule, not a robots.txt line.
- What does HTTP 403 mean in the results?
- Your server refused the request. That is separate from robots.txt and often comes from CDN or WAF bot protection rather than anything you configured directly. It is important to read it carefully: because our request does not come from the crawler’s verified infrastructure, a 403 to us does not prove the genuine crawler is blocked. A site that correctly rejects unverified claimants will refuse us and serve the real crawler normally.
- Does passing the check mean AI systems will mention my website?
- No. Access is a precondition, not a cause. A crawler being able to fetch your page does not mean it will fetch it, index it, retrieve it for a given question, or cite it in an answer. Anyone offering you a probability for that is guessing.
- Does Outranked guarantee AI visibility?
- No, and we will not claim otherwise. Our about page lists “claim that paying us improves your Google ranking or gets you mentioned by an AI assistant” among the things this business refuses to say. Outranked is a public ranking based on verified spend; it is unrelated to what this tool measures.
How we identify ourselves
When the checker requests your page as a given crawler it sends that crawler’s user-agent, because otherwise it could not observe how your server treats it. Every other request Outranked makes identifies itself as Outranked. What we fetch, what we never do and how to block us are on the crawler page. Other free checks live under tools.