Robots.txt Checker

Enter a domain to see what its robots.txt file tells Google, Bing, ChatGPT, Claude, Perplexity and other AI crawlers...

What this tool tells you

Every website can publish a small text file at /robots.txt that tells crawlers which parts of the site they may visit. It’s easy to get wrong, and a single line can remove a newsroom from Google or from AI answers. This checker reads the file and explains it in plain English, in sections:

• At a glance: whether Google, AI search tools and AI training crawlers can read the site.
• What to check: problems and warnings, most serious first.
• The file itself: whether it loads, redirects or errors, and what search engines do in each case.
• Sitemaps: which sitemaps are listed and whether they load.
• Search engines, AI search, AI assistants and AI training: what each named bot is allowed to do, and what a block means for your audience.
• Test a page: paste an article address to see which bots may read it.
• Line by line: every line of the file, translated.

Robots.txt basics

Crawling means a bot visiting a page to read it. Search engines and AI tools need to crawl a page before they can use what’s on it. Robots.txt tells them which pages they may crawl.

It controls crawling, not indexing. A page blocked in robots.txt can still show up in Google if other sites link to it, just without a description. To keep a page out of search results, let Google crawl it and add a noindex tag.

It isn’t security. The file is public and anyone can read it. Well-behaved bots follow it, but nothing forces them to. Don’t use it to hide anything sensitive.

The most specific section wins. If the file has rules for all bots (User-agent: *) and a separate section for, say, GPTBot, then GPTBot follows only its own section and ignores the rules for all bots. This catches out a lot of publishers.

AI training and AI search are different bots. OpenAI, Anthropic and Perplexity each use one crawler for search and citations and, where they train models, another for training. You can block training and still be cited in AI answers.

Google-Extended doesn’t control AI Overviews. It covers Gemini training and grounding in the Gemini app and Vertex AI. AI Overviews and AI Mode use Googlebot, the same crawler as Search.

Each host has its own file. www.example.com, example.com and news.example.com are checked separately.

Need help with it?

If the report raises questions, I run technical SEO audits and AI visibility work for publishers, and training that explains this for editorial teams. Book a call or get in touch.