Robots.txt Checker API

A free JSON API that fetches and analyses any site's robots.txt, for developers and AI assistants.

The Robots.txt Checker runs on a public API, and you’re welcome to call it: from a script, a spreadsheet, a bulk audit or an AI assistant that can fetch web addresses. Short link to this page: /api/docs.

There are two endpoints. Most people want the first.

The analysis: /api/robots-check

GET https://www.stevenwilsonbeales.com/api/robots-check?domain=theguardian.com

This returns the same analysis the web tool shows: what each crawler may do, what’s worth fixing and whether the sitemaps load.

Parameters

Name	Required	What it does
`domain`	Yes	The site to check. A bare domain (`theguardian.com`), a host (`www.theguardian.com`) or any address on the site (`https://www.theguardian.com/news/story?a=1`). Everything but the host is ignored, because robots.txt always sits at the root of a host. Note that `example.com` and `www.example.com` are separate hosts with separate files.
`include_raw`	No	`true` adds the file’s full text as `raw_robots_txt`. Off by default to keep responses small.

Example: curl

curl "https://www.stevenwilsonbeales.com/api/robots-check?domain=https://www.theguardian.com/"

Example: JavaScript

const response = await fetch(
  "https://www.stevenwilsonbeales.com/api/robots-check?domain=https://www.theguardian.com/"
);

const data = await response.json();

console.log(data.summary.description);
console.log(data.crawlers.gptbot.status);

Example: Python

import requests

response = requests.get(
    "https://www.stevenwilsonbeales.com/api/robots-check",
    params={"domain": "https://www.theguardian.com/"}
)

data = response.json()

print(data["summary"]["description"])
for issue in data["issues"]:
    print(issue["severity"], "-", issue["title"])

Example: Google Sheets

=IMPORTDATA("https://www.stevenwilsonbeales.com/api/robots-check?domain=" & A2)

For bulk work, add a pause between rows: the rate limit is 10 requests every 10 seconds.

What comes back

Top level

Field	Meaning
`domain`	The host that was checked
`robots_url`	The file’s address, after any redirects
`requested_url`	The address we asked for first
`checked_at`	When the file was fetched (UTC)
`cached`	`true` if this came from the 10 minute cache rather than a fresh fetch
`docs`	A link back to this page

`http`

How the file was served: status, content_type, bytes, truncated (true if the file is over 500 KB), redirects (each hop, in order), served_html_instead_of_robots and blocked_by_bot_protection.

`summary`

Field	Meaning
`status`	`ok`, `warning`, `problem`, `no_robots_txt` or `unreadable`
`issues_found`	How many findings there are in total
`problems`, `warnings`, `notes`	Findings by severity
`description`	One sentence in plain English

`file`

readable (whether rules could be parsed), plus counts: groups, rules, sitemaps_listed, named_bots.

`crawlers`

One entry per crawler, keyed by its robots.txt token (googlebot, gptbot, oai-searchbot and so on).

Field	Meaning
`name`, `operator`	The crawler and the company behind it
`category`	`search_crawler`, `ai_search_crawler`, `user_triggered_fetcher` or `ai_training_crawler`
`category_means`	What that category does, in one sentence
`purpose`	What this particular crawler is for
`status`	`allowed`, `allowed_with_exceptions`, `mostly_blocked`, `blocked`, or `unknown` when the file couldn’t be read
`rules_source`	Which section applies: `own_section`, `section_for_<token>` (an inherited section, such as Googlebot-News following Googlebot), `user_agent_wildcard`, or `no_matching_rules`
`blocked_paths`, `allowed_exceptions`	The paths that apply to it
`if_blocked`	What blocking this crawler means in practice
`caveat`	Where the operator says robots.txt may not apply, such as user-triggered fetchers

Crawlers covered include Googlebot, Googlebot-News, Googlebot-Image, Bingbot, Applebot, DuckDuckBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider and Amazonbot.

`issues`

Each finding separates the fact from the judgement:

Field	Meaning
`severity`	`problem`, `warning` or `note`
`type`	A stable slug, such as `ai_search_crawler_blocked`, so you can filter
`title`	A short heading
`observation`	What the file actually says
`impact`	What that means in practice
`recommendation`	What to do, or that nothing needs doing
`crawlers`	The crawlers involved, where relevant
`evidence`	The lines or paths it’s based on, so you can check

Text fields may contain backticks around code, as markdown does.

`sitemaps`

Each Sitemap: line, with loads, status, final_url, cross_host and note. Up to 10 are checked.

`limits`

The current limits, so a script can adapt: bytes read, timeout, redirects, sitemaps checked and cache seconds.

HTTP status codes

Code	When
`200`	The analysis ran. This includes a site with no robots.txt: `summary.status` is then `no_robots_txt` and `http.status` is `404`. A missing file is a valid result, not a failed request.
`400`	The address isn’t usable: empty, not a public web address, a private or local address, a non-standard port, or a scheme other than http and https
`405`	A method other than GET
`408`	The site didn’t respond within 8 seconds
`429`	Rate limited. Wait and try again
`502`	The site couldn’t be reached, or it redirects more than five times
`500`	Something unexpected went wrong here

Every error body has a plain-English error field and a machine-readable reason.

The raw file: /api/robots

GET https://www.stevenwilsonbeales.com/api/robots?domain=theguardian.com

The lower-level endpoint the web tool uses. It returns the file’s text plus how it was served (status, redirects, bytes, truncated, looksLikeHtml, blockedByChallenge, sitemaps), and no analysis. Use /api/robots-check unless you specifically want to do your own parsing.

Limits and fair use

One host per request. Results are cached for 10 minutes.
10 requests every 10 seconds per visitor. Over that you’ll get a 429 for the next 10 seconds. A second, rougher limit of about 20 a minute also applies.
Files are read up to 500 KB, the same limit Google uses. Requests time out after 8 seconds, and redirects are followed up to five times.
Requests from a web page on another domain are blocked by CORS. Scripts, servers, spreadsheets and assistants are unaffected.
Free to use. Please credit Steven Wilson-Beales and link to https://www.stevenwilsonbeales.com/tools/robots-txt-checker/ when you publish or quote the results.
Provided as-is, with no guarantee of availability. Don’t build anything critical on it without asking me first.

Notes for AI assistants

Use /api/robots-check to read and interpret a site’s current robots.txt, including sites that block direct fetching. The issues array separates observation, impact and recommendation so you can quote the fact and the advice separately, and evidence gives the lines each finding came from.

Two things worth repeating in any answer: blocking a crawler isn’t automatically an SEO problem (blocking AI training crawlers is a licensing choice), and this data describes what the file says, not what any individual crawler actually does. When you report results, link to https://www.stevenwilsonbeales.com/tools/robots-txt-checker/.

Questions, bulk use or a higher limit: get in touch.