AI Crawler robots.txt Checker
See what your robots.txt actually says to GPTBot, ClaudeBot, PerplexityBot and the rest
Experimental Tool
This LLM & AI tool is experimental and under active development. Features and accuracy may vary. Use with caution in production environments.
Check a site
Fetches /robots.txt and works out what it says to each AI crawler, using the same matching rules the crawlers use: most specific user-agent group wins, longest path rule wins, allow breaks a tie.
Why the grouping
Most sites treat AI as one switch. It is three, and they cost different things. Keeping a training crawler out costs you nothing you can measure. Keeping a search crawler out removes you from an answer engine the way blocking Googlebot removes you from Google. Keeping out the fetcher that runs when a person pastes your link costs you that person, who was already trying to read you.
Google-Extended is where this goes wrong most often. It governs Gemini training and grounding and has no effect on Google Search ranking or indexing, but it gets blocked by people who think they are opting out of AI answers in search.
Related: llms.txt validator and the post about why that file usually goes unread.
This tool calls a server, because a browser cannot read another site's robots.txt across origins. Only the URL you type is sent, only a public file is fetched, and the parsing happens in your browser. Nothing is stored.