Block the Bot?

Crawls pages for Siri and Spotlight suggestions and Safari search features.

Block it? It sends you search visitors. Block only areas you don't want in search, such as account pages.

In robots.txt: User-agent: Applebot #

Crawls pages for Bing and for Microsoft Copilot's web answers. Bing's index also supplies results to other search services.

Block it? It sends you search visitors. Block only areas you don't want in search, such as account pages.

In robots.txt: User-agent: Bingbot #

DuckDuckGo's own crawler, used alongside results from partners.

Block it? It sends you search visitors. Block only areas you don't want in search, such as account pages.

In robots.txt: User-agent: DuckDuckBot #

Crawls pages for Google Search, including AI Overviews and AI Mode, and for Google Discover.

Block it? It sends you search visitors. Block only areas you don't want in search, such as account pages.

In robots.txt: User-agent: Googlebot #

Crawls images for Google Images and image results in Search.

Block it? It sends you search visitors. Block only areas you don't want in search, such as account pages.

In robots.txt: User-agent: Googlebot-Image #

Crawls articles for Google News.

Block it? It sends you search visitors. Block only areas you don't want in search, such as account pages.

In robots.txt: User-agent: Googlebot-News #

Indexes pages to improve the search results Claude uses in its answers.

Block it? Claude is less likely to find and cite your pages when it searches the web.

In robots.txt: User-agent: Claude-SearchBot #

Finds pages to show and link to in ChatGPT search results. OpenAI says it isn't used for training, which GPTBot controls, but if you allow both bots OpenAI may use one crawl for both purposes.

Block it? Your pages are unlikely to be shown or linked as sources in ChatGPT search answers.

In robots.txt: User-agent: OAI-SearchBot #

Finds pages to show and link to in Perplexity answers. Perplexity says it isn't used to train AI models.

Block it? Your pages are unlikely to appear as sources in Perplexity.

In robots.txt: User-agent: PerplexityBot #

Visits a page when a ChatGPT user asks about it or ChatGPT needs it to answer a question.

Block it? ChatGPT may be unable to open your pages for its users.

OpenAI says robots.txt rules may not apply, because a person started the request.

In robots.txt: User-agent: ChatGPT-User #

Visits a page when a Claude user asks a question that needs it.

Block it? Claude can't open your pages for its users, which may reduce how often you're cited.

In robots.txt: User-agent: Claude-User #

Visits a page when a Perplexity user's question needs it.

Block it? Perplexity says it generally ignores robots.txt, so a block may have little effect.

Perplexity says this fetcher generally ignores robots.txt.

In robots.txt: User-agent: Perplexity-User #

Not a bot. A shared signal, proposed in October 2026, that tells AI companies whether pages may be used to train AI models. It doesn't control crawling, and the rules for all bots don't count for it.

Use it? It asks AI companies not to train on your pages without blocking any bot. It doesn't affect search.

A rule that names a company's own bot comes first. Anthropic follows a ClaudeBot rule over this signal, and Google only applies the signal when the file doesn't name Google-Extended.

In robots.txt: User-agent: __usage_ai-training #

Crawls pages to improve Amazon products such as Alexa's answers, and may be used to train Amazon's AI models.

Block it? Alexa and other Amazon services can't use your pages.

In robots.txt: User-agent: Amazonbot #

Not a separate crawler: a switch that tells Apple whether pages Applebot crawls may be used to train its AI models.

Block it? Apple won't use your pages to train its models. Siri and Spotlight can still show them.

In robots.txt: User-agent: Applebot-Extended #

ByteDance's crawler (TikTok's owner), linked to its AI products.

Block it? Asks ByteDance not to crawl. Publishers have reported it doesn't always comply.

In robots.txt: User-agent: Bytespider #

Builds Common Crawl, a free public archive of the web that many AI companies use as training data.

Block it? New pages stay out of future Common Crawl archives. Copies already in older archives remain.

In robots.txt: User-agent: CCBot #

Collects pages that may be used to train Anthropic's Claude models.

Block it? Anthropic excludes your future content from its training data. It doesn't affect Claude's search or user fetches.

In robots.txt: User-agent: ClaudeBot #

Not a separate crawler: a switch that tells Google whether pages Googlebot crawls may be used to train Gemini, and to ground answers in the Gemini app and Vertex AI.

Block it? Your content isn't used for Gemini training or grounding. It does not remove you from Google Search, AI Overviews or AI Mode, and isn't a ranking signal.

In robots.txt: User-agent: Google-Extended #

Collects pages that may be used to train OpenAI's AI models.

Block it? OpenAI treats this as a request not to use your content to train its models. It doesn't affect ChatGPT search.

In robots.txt: User-agent: GPTBot #

Collects pages for Meta's AI work, such as training models.

Block it? Meta shouldn't collect your pages for AI training.

In robots.txt: User-agent: Meta-ExternalAgent #

Builds an index of web pages for AI agents to read. Lyrenth says it follows robots.txt.

Block it? It helps AI tools find pages to cite. Blocking it makes Lyrenth's answers less likely to include or link to you.

In robots.txt: User-agent: AIWebIndex #

Finds pages for search features in Amazon products such as Alexa. Amazon says it isn't used to train AI models.

Block it? It helps AI tools find pages to cite. Blocking it makes Amazon's answers less likely to include or link to you.

In robots.txt: User-agent: Amzn-SearchBot #

Crawls pages so they can appear in Direqt's AI search results. Direqt says it follows robots.txt and doesn't use the content to train models.

Block it? It helps AI tools find pages to cite. Blocking it makes Direqt's answers less likely to include or link to you.

In robots.txt: User-agent: Anomura #

Crawls websites that Atlassian customers connect to Rovo, so the content shows up in Rovo search and chat. It only crawls a site whose robots.txt has a rule for it.

Block it? It only crawls sites that Atlassian's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: atlassian-bot #

Crawls pages for Baidu, China's biggest search engine.

Block it? Blocking it keeps your pages out of Baidu's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Baiduspider #

Baidu's crawler for its image search.

Block it? Blocking it keeps your pages out of Baidu's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Baiduspider-image #

Baidu's crawler for its video search.

Block it? Blocking it keeps your pages out of Baidu's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Baiduspider-video #

Rendered pages so Bing could show a preview of a result. Microsoft no longer documents it, and it may now be part of Bingbot.

Block it? An old or unused name, so a rule for it probably does nothing. Use the bot's current name instead.

In robots.txt: User-agent: BingPreview #

Listed by bot directories as the crawler for Brave Search. Brave's own help page says its crawler doesn't use a name of its own, so this rule may do nothing.

Block it? Brave says its crawler doesn't use this name, so a rule for it probably does nothing.

In robots.txt: User-agent: Bravebot #

Crawls product pages for Channel3's product catalogue, which AI shopping agents search. Channel3 says you can control it with robots.txt.

Block it? AI shopping agents search its product catalogue. Blocking it keeps your products out of their results.

In robots.txt: User-agent: Channel3Bot #

The crawler for Cliqz, a search engine and browser that announced its closure in April 2020. The rule probably does nothing now.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: Cliqzbot #

Covers every crawler from Cốc Cốc, a Vietnamese search engine. A rule for it applies to coccocbot-web and the others.

Block it? Blocking it keeps your pages out of Cốc Cốc's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: coccocbot #

Crawls pages for Cốc Cốc, a Vietnamese search engine.

Block it? Blocking it keeps your pages out of Cốc Cốc's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: coccocbot-web #

Indexes pages for Daum Search, a South Korean portal.

Block it? Blocking it keeps your pages out of Daum's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Daumoa #

Fetches pages for DuckDuckGo's AI-assisted answers. DuckDuckGo says it doesn't use them to train AI models.

Block it? It opens a page when someone asks DuckDuckGo's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: DuckAssistBot #

Crawled for Exalead, a French search engine that has closed, so this rule no longer does anything. Not the same as Exa's ExaSearchBot.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: Exabot #

Fetches images on pages ExaSearchBot has crawled, for thumbnails in Exa's search results. Without its own rules it follows ExaSearchBot's.

Block it? It helps AI tools find pages to cite. Blocking it makes Exa's answers less likely to include or link to you.

In robots.txt: User-agent: ExaSearchBot-Images #

The crawler for Gigablast, a search engine that went offline in April 2023. The rule probably does nothing now.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: Gigabot #

Used by Search Console testing tools, such as URL Inspection. Blocking it stops those tools testing your pages.

Block it? Search Console's testing tools use it. Blocking it stops you testing your own pages.

In robots.txt: User-agent: Google-InspectionTool #

Fetches a page so Google can read it aloud to someone who asked. Being user-triggered, it generally ignores robots.txt.

Block it? It fetches a page only when someone asks Google to read it aloud, and generally ignores robots.txt.

In robots.txt: User-agent: Google-Read-Aloud #

Checks links for malware and abuse. Google says it ignores robots.txt, so rules for it have no effect.

Block it? Google says it ignores robots.txt, so a rule for it has no effect.

In robots.txt: User-agent: Google-Safety #

Google's old crawler for pages built for feature phones. Google's current crawler list doesn't include it, so the rule probably does nothing now.

Block it? Google no longer lists this crawler, so the rule probably does nothing. Googlebot covers mobile pages.

In robots.txt: User-agent: Googlebot-Mobile #

Crawls videos for video results in Google Search.

Block it? It puts your videos in Google Search. Blocking it removes them from video results.

In robots.txt: User-agent: Googlebot-Video #

Google's crawler for research and product development. It isn't used for Google Search.

Block it? Google says it isn't used for Search, so blocking it doesn't affect your rankings.

In robots.txt: User-agent: GoogleOther #

Google's general-purpose crawler, fetching images. Google says only that various product teams may use it, so treating it as an AI crawler is guesswork.

Block it? Google says it isn't used for Search, so blocking it doesn't affect your rankings.

In robots.txt: User-agent: GoogleOther-Image #

Google's general-purpose crawler, fetching videos. Google says only that various product teams may use it.

Block it? Google says it isn't used for Search, so blocking it doesn't affect your rankings.

In robots.txt: User-agent: GoogleOther-Video #

Crawls pages for Ibou, a search engine. Ibou says it follows robots.txt and doesn't train AI models on the data.

Block it? Blocking it keeps your pages out of Ibou's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: IbouBot #

Crawls pages for the Kagi search engine. Kagi says it follows rules aimed at Kagibot, and Googlebot's rules if there are none.

Block it? Blocking it keeps your pages out of Kagi's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Kagibot #

Crawls pages for Keenable's search index, which AI agents and AI companies query.

Block it? It helps AI tools find pages to cite. Blocking it makes Keenable's answers less likely to include or link to you.

In robots.txt: User-agent: KeenableBot #

Builds the search index behind Kimi, Moonshot AI's assistant. Blocking it keeps your pages out of Kimi's search results.

Block it? It helps AI tools find pages to cite. Blocking it makes Moonshot AI's answers less likely to include or link to you.

In robots.txt: User-agent: Kimi-SearchBot #

Crawls pages for Linkup's web search index, which supplies search results to AI applications. Linkup says it follows robots.txt.

Block it? It helps AI tools find pages to cite. Blocking it makes Linkup's answers less likely to include or link to you.

In robots.txt: User-agent: LinkupBot #

Builds the search index Mistral's assistant uses to answer questions. Mistral says the content isn't used to train AI models.

Block it? It helps AI tools find pages to cite. Blocking it makes Mistral AI's answers less likely to include or link to you.

In robots.txt: User-agent: MistralAI-Index #

Crawls pages for Mojeek, an independent UK search engine with its own index.

Block it? Blocking it keeps your pages out of Mojeek's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Mojeek #

Crawls pages for Mojeek, an independent UK search engine with its own index.

Block it? Blocking it keeps your pages out of Mojeek's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: MojeekBot #

Microsoft's old search crawler, replaced by Bingbot.

Block it? Replaced by Bingbot. To control Bing, use a Bingbot rule.

In robots.txt: User-agent: msnbot #

Crawled pages for Neeva, an ad-free search engine that shut down in June 2023. The rule probably does nothing now.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: Neevabot #

Collects headlines for NewsNow, a UK news aggregator.

Block it? Blocking it keeps your pages out of NewsNow's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: NewsNow #

Crawls pages for Petal Search, Huawei's search engine.

Block it? Blocking it keeps your pages out of Huawei's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: PetalBot #

Fetches pages for Phind, an AI answer engine for developers.

Block it? It helps AI tools find pages to cite. Blocking it makes Phind's answers less likely to include or link to you.

In robots.txt: User-agent: PhindBot #

Crawls pages for Gist.ai, ProRata's AI answer service. ProRata says it follows robots.txt.

Block it? It helps AI tools find pages to cite. Blocking it makes ProRata.ai's answers less likely to include or link to you.

In robots.txt: User-agent: ProRataInc #

Listed by bot directories as indexing pages for Querit's search service, which AI applications query. Querit doesn't document it.

Block it? It helps AI tools find pages to cite. Blocking it makes Querit's answers less likely to include or link to you.

In robots.txt: User-agent: Querit-SearchBot #

Crawls pages for Qwant, a French search engine with its own index, and for Qwant News (as Qwantbot-news).

Block it? Blocking it keeps your pages out of Qwant's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Qwantbot #

Crawls pages for Seekport, a search engine run by the German SEO company SISTRIX. SISTRIX says it follows robots.txt.

Block it? Blocking it keeps your pages out of SISTRIX's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: SeekportBot #

Crawls pages for Seznam, a Czech search engine.

Block it? Blocking it keeps your pages out of Seznam's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: SeznamBot #

Fetches articles and media for the Seznam.cz home page, separately from SeznamBot.

Block it? Blocking it keeps your pages out of Seznam.cz's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: SeznamHomepageCrawler #

Yahoo's old search crawler. Yahoo's search results now mostly come from Bing.

Block it? Yahoo's results now mostly come from Bing, so blocking Slurp costs little. Don't block Bingbot.

In robots.txt: User-agent: Slurp #

Aimed at the crawlers of Sogou, a Chinese search engine. Sogou says its crawler follows robots.txt.

Block it? Blocking it keeps your pages out of Sogou's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Sogou #

Crawls pages for Sogou, a Chinese search engine.

Block it? Blocking it keeps your pages out of Sogou's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Sogou web spider #

Crawled for Soso, a Chinese search engine folded into Sogou in 2013. Defunct.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: Sosospider #

Crawls product pages for Google Shopping.

Block it? It puts your products in Google Shopping. Only block it if you don't sell anything.

In robots.txt: User-agent: Storebot-Google #

Crawls pages for Timpi's decentralised search index, which is also sold as a web data service.

Block it? Blocking it keeps your pages out of Timpi's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Timpibot #

Listed by bot directories as indexing pages that may be cited in Grok's AI search answers. xAI doesn't document it.

Block it? It helps AI tools find pages to cite. Blocking it makes xAI's answers less likely to include or link to you.

In robots.txt: User-agent: xAI-SearchBot #

Free, open-source search software that anyone can run, so these requests can come from many different people.

Block it? Anyone can run this search software, and it sends little traffic. Blocking it is low risk.

In robots.txt: User-agent: YaCy #

Covers all of Yandex's bots, including its search crawler.

Block it? Blocking it keeps your pages out of Yandex's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Yandex #

Not a separate bot. It tells Yandex whether pages it has already indexed can be used in its AI answers.

Block it? It tells Yandex whether your pages can be used in its AI answers. It doesn't affect Yandex search.

In robots.txt: User-agent: YandexAdditional #

Not a separate bot. It tells Yandex whether pages it has already indexed can be used in its AI answers.

Block it? It tells Yandex whether your pages can be used in its AI answers. It doesn't affect Yandex search.

In robots.txt: User-agent: YandexAdditionalBot #

Crawls pages for Yandex, Russia's biggest search engine.

Block it? Blocking it keeps your pages out of Yandex's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: YandexBot #

Indexes images so they can appear in Yandex Images.

Block it? Blocking it keeps your pages out of Yandex's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: YandexImages #

Crawls pages for Naver, South Korea's biggest search engine.

Block it? Blocking it keeps your pages out of Naver's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: Yeti #

Crawls pages for the Yisou search engine.

Block it? Blocking it keeps your pages out of Yisou's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: YisouSpider #

Crawls pages for You.com's AI search and answers.

Block it? It helps AI tools find pages to cite. Blocking it makes You.com's answers less likely to include or link to you.

In robots.txt: User-agent: YouBot #

Crawls public web pages for Zanista's search platform.

Block it? Blocking it keeps your pages out of Zanista's search results. Little is lost if you get no visitors from there.

In robots.txt: User-agent: ZanistaBot #

Collects pages to train open-source AI models.

Block it? An AI company's bot. Block it if you don't want Ai2 (Allen Institute for AI) using your content.

In robots.txt: User-agent: AI2Bot #

Fetches pages used in deep research queries run by the open-source AI models of Ai2, a non-profit research institute.

Block it? It opens a page when someone asks Ai2's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: AI2Bot-DeepResearchEval #

Collects pages for Dolma, an open dataset used to train AI models.

Block it? An AI company's bot. Block it if you don't want Ai2 (Allen Institute for AI) using your content.

In robots.txt: User-agent: Ai2Bot-Dolma #

The crawler for Amazon Kendra, a search service that AWS customers point at websites to build their own search tools. Amazon says it follows robots.txt.

Block it? It only crawls sites that Amazon Web Services' customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: amazon-kendra #

Used by Amazon Q Business, which companies use to build their own AI assistants from web pages they choose.

Block it? It only crawls sites that Amazon's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: Amazon-QBusiness #

The agent for Amazon's Buy for Me service. It visits product pages on other shops to gather product details and place orders for Amazon customers.

Block it? It buys products from other shops for Amazon customers. If you sell online, blocking it stops those sales.

In robots.txt: User-agent: AmazonBuyForMe #

Fetches a page when someone's request needs it, such as an Alexa question. Amazon says it isn't used to train AI models.

Block it? It opens a page when someone asks Amazon's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: Amzn-User #

The name of Andi's original AI assistant. Andi says a rule for it applies to Andi's AI answers, not to its search crawler, Andi-SearchBot.

Block it? A rule for it applies to Andi's AI answers. Blocking it makes Andi less likely to use or cite your pages.

In robots.txt: User-agent: AndiBot #

An old name. Anthropic's current bots are ClaudeBot, Claude-SearchBot and Claude-User, so this rule probably does nothing on its own.

Block it? An old name. To control Anthropic, use ClaudeBot, Claude-SearchBot or Claude-User.

In robots.txt: User-agent: anthropic-ai #

Fetches website content for Big Sur AI, which builds AI sales assistants and web agents for businesses.

Block it? It only crawls sites that Big Sur AI's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: bigsur.ai #

Collects pages to train and test the ChatGLM language models.

Block it? An AI company's bot. Block it if you don't want Zhipu AI using your content.

In robots.txt: User-agent: ChatGLM-Spider #

OpenAI's ChatGPT agent, which uses a web browser to carry out tasks for a person. OpenAI's own bot page doesn't list this as a robots.txt name, so the rule may do nothing.

Block it? OpenAI doesn't list this as a robots.txt name, so the rule may do nothing.

In robots.txt: User-agent: ChatGPT Agent #

A browser Anthropic hosts so Claude products, such as Claude Duet, can carry out tasks a person asks for. Anthropic says it's separate from its crawlers and doesn't say whether it follows robots.txt.

Block it? An AI agent acting for a person, such as booking or shopping. Blocking it stops those users completing tasks on your site.

In robots.txt: User-agent: Claude-Agent #

Claude Code, Anthropic's coding tool, fetches pages as Claude-User, so a rule for this name probably does nothing. Use Claude-User instead.

Block it? Claude Code fetches pages as Claude-User, so use that name instead.

In robots.txt: User-agent: Claude-Code #

An old name. Anthropic's current bots are ClaudeBot, Claude-SearchBot and Claude-User, so this rule probably does nothing on its own.

Block it? An old name. To control Anthropic, use ClaudeBot, Claude-SearchBot or Claude-User.

In robots.txt: User-agent: Claude-Web #

Listed by bot directories as an agent that logs in to software providers to fetch invoices and manage users for Cledara customers. It isn't expected to follow robots.txt.

Block it? It only crawls sites that Cledara's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: CledaraBot #

The old name of the crawler for Cloudflare AI Search, which indexes websites that Cloudflare customers connect. Cloudflare renamed it Cloudflare-AI-Search in February 2026 but still accepts the old name in robots.txt.

Block it? An old name. Cloudflare now uses Cloudflare-AI-Search.

In robots.txt: User-agent: Cloudflare-AutoRAG #

A short form of Google-CloudVertexBot, which crawls a site when its owner asks Google to, for building Vertex AI agents. Google documents only the longer name, so this rule may not match.

Block it? Google documents only the longer name, Google-CloudVertexBot, so this rule may not match.

In robots.txt: User-agent: CloudVertexBot #

Linked to Cohere, an AI company that builds language models for businesses. Cohere publishes little about this token.

Block it? An AI company's bot. Block it if you don't want Cohere using your content.

In robots.txt: User-agent: cohere-ai #

Collects pages that may be used to train Cohere's language models.

Block it? An AI company's bot. Block it if you don't want Cohere using your content.

In robots.txt: User-agent: cohere-training-data-crawler #

Collects Japanese-language pages for a national research data project.

Block it? An AI company's bot. Block it if you don't want ROIS-DS (Japan) using your content.

In robots.txt: User-agent: Cotoyogi #

Reported to collect pages for DeepSeek's models. DeepSeek publishes nothing confirming this crawler exists.

Block it? An AI company's bot. Block it if you don't want DeepSeek using your content.

In robots.txt: User-agent: DeepSeekBot #

Devin, an AI software engineering agent that browses websites while doing coding tasks for its users.

Block it? An AI agent acting for a person, such as booking or shopping. Blocking it stops those users completing tasks on your site.

In robots.txt: User-agent: Devin #

Extracts facts from web pages into a large database that other companies use, including for AI.

Block it? An AI company's bot. Block it if you don't want Diffbot using your content.

In robots.txt: User-agent: Diffbot #

Fetches a single page when a person using Diffbot asks for it, rather than crawling on its own.

Block it? It opens a page when someone asks Diffbot's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: Diffbot-User #

Listed by bot directories as collecting pages to train AI models for Doubao, ByteDance's assistant. ByteDance publishes nothing confirming it.

Block it? An AI company's bot. Block it if you don't want ByteDance using your content.

In robots.txt: User-agent: Doubaobot #

Fetches pages that ElevenLabs customers add to their AI agents' knowledge bases, and again when they refresh them. ElevenLabs says it follows robots.txt.

Block it? It only crawls sites that ElevenLabs' customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: ElevenlabsBot #

Collects pages that may be used to train ElevenLabs' AI models. ElevenLabs says blocking it means your content shouldn't be used for training.

Block it? An AI company's bot. Block it if you don't want ElevenLabs using your content.

In robots.txt: User-agent: ElevenlabsCrawler #

Listed by bot directories as collecting pages to train Baidu's ERNIE AI models. Baidu publishes nothing confirming it.

Block it? An AI company's bot. Block it if you don't want Baidu using your content.

In robots.txt: User-agent: ERNIEBot #

Builds the index behind Exa's search API, which AI applications use to find sources.

Block it? It helps AI tools find pages to cite. Blocking it makes Exa's answers less likely to include or link to you.

In robots.txt: User-agent: ExaSearchBot #

Fetches a page live when a person, or an app built on Exa's search API, asks for its current content. Exa says robots.txt rules may not apply, because a user started the fetch.

Block it? It opens a page when someone asks Exa's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: ExaUser #

Meta has said it crawls public pages to improve the language models behind its speech recognition.

Block it? An AI company's bot. Block it if you don't want Meta using your content.

In robots.txt: User-agent: FacebookBot #

Says it collects pages to build machine-learning datasets. No company has publicly claimed it.

Block it? An AI company's bot. Block it if you don't want Unclaimed using your content.

In robots.txt: User-agent: FriendlyCrawler #

Fetches pages when someone asks Gemini's Deep Research feature to look into a topic.

Block it? It opens a page when someone asks Google's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: Gemini-Deep-Research #

Used by AI agents running on Google's systems to browse the web and take actions when a person asks. Google says fetchers started by a user generally ignore robots.txt.

Block it? An AI agent acting for a person. Google says user-started fetchers generally ignore robots.txt.

In robots.txt: User-agent: Google-Agent #

Crawls a site when its owner builds an AI agent with Google Cloud's Vertex AI. It only matters if your organisation uses that.

Block it? It only crawls sites that Google's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: Google-CloudVertexBot #

The old name of the fetcher that loads pages people add as sources in Google's notebook tool. Google supported it until August 2026 and now uses Google-GeminiNotebook. Google says fetchers started by a user generally ignore robots.txt.

Block it? An old name. Google now uses Google-GeminiNotebook.

In robots.txt: User-agent: Google-NotebookLM #

Google's Project Mariner, an AI agent that uses a web browser to carry out tasks for a person. Google's own crawler pages don't list this name.

Block it? An AI agent acting for a person, such as booking or shopping. Blocking it stops those users completing tasks on your site.

In robots.txt: User-agent: GoogleAgent-Mariner #

Listed by bot directories as fetching pages for developers using the Gemini API's URL feature. Google's own crawler pages don't list this name.

Block it? It opens a page when someone asks Google's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: GoogleAgent-URLContext #

Listed by bot directories as an xAI crawler. xAI publishes nothing saying what it collects pages for.

Block it? An AI company's bot. Block it if you don't want xAI using your content.

In robots.txt: User-agent: GrokBot #

Listed by bot directories as collecting pages to train Tencent's Hunyuan AI models. Tencent publishes nothing confirming it.

Block it? An AI company's bot. Block it if you don't want Tencent using your content.

In robots.txt: User-agent: Hunyuan #

Identifies itself as a crawler from iAsk, an AI answer service. iAsk publishes nothing saying what it collects pages for.

Block it? An AI company's bot. Block it if you don't want iAsk using your content.

In robots.txt: User-agent: iaskspider #

Collects pages for a Japanese national research institute, for machine translation and analysis research. Content collected since 2024 may also be shared with partners.

Block it? An AI company's bot. Block it if you don't want NICT (Japan) using your content.

In robots.txt: User-agent: ICC-Crawler #

Collects images for Hive's image search and content moderation services.

Block it? An AI company's bot. Block it if you don't want Hive using your content.

In robots.txt: User-agent: ImageSift #

Collects images for Hive's image search and content moderation services.

Block it? An AI company's bot. Block it if you don't want Hive using your content.

In robots.txt: User-agent: ImagesiftBot #

Reported to collect images from websites for ByteDance, the company behind TikTok. ByteDance publishes nothing confirming it.

Block it? An AI company's bot. Block it if you don't want ByteDance using your content.

In robots.txt: User-agent: imageSpider #

Listed by bot directories as fetching pages when someone uses Kagi's AI assistant. Kagi's own bot page covers only Kagibot.

Block it? It opens a page when someone asks Kagi's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: kagi-fetcher #

Collects Australian web pages to train an open-source Australian language model.

Block it? An AI company's bot. Block it if you don't want Kangaroo LLM using your content.

In robots.txt: User-agent: Kangaroo Bot #

Listed by bot directories as a browser agent that carries out tasks for Kimi users. Moonshot doesn't document it, and it isn't expected to follow robots.txt.

Block it? An AI agent acting for a person, such as booking or shopping. Blocking it stops those users completing tasks on your site.

In robots.txt: User-agent: Kimi-Agent #

Fetches a page when a Kimi user asks for it, for example to summarise an article. Moonshot says robots.txt rules may not apply, because a person started the request.

Block it? It opens a page when someone asks Kimi about it. Moonshot says robots.txt may not apply, because a person started the request.

In robots.txt: User-agent: Kimi-User #

Collects pages that may be used to train the models behind Kimi, Moonshot AI's assistant.

Block it? An AI company's bot. Block it if you don't want Moonshot AI using your content.

In robots.txt: User-agent: KimiBot #

Fetches pages from websites that Klaviyo customers have connected to their accounts, to power Klaviyo's AI features. Klaviyo says it follows robots.txt.

Block it? It only crawls sites that Klaviyo's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: KlaviyoAIBot #

Reported to be a crawler from LAION, a non-profit that builds open datasets for AI research. LAION publishes nothing confirming it.

Block it? An AI company's bot. Block it if you don't want LAION using your content.

In robots.txt: User-agent: laion-huggingface-processor #

Fetches a page when a Liner user asks a question, and cites it as a source. Rules for LinerBot don't apply to it.

Block it? It opens a page when someone asks Liner's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: Liner-User #

Indexes pages for Liner, an AI search engine that answers with citations.

Block it? It helps AI tools find pages to cite. Blocking it makes Liner's answers less likely to include or link to you.

In robots.txt: User-agent: LinerBot #

Manus, an AI agent that uses a web browser to carry out tasks for its users.

Block it? An AI agent acting for a person, such as booking or shopping. Blocking it stops those users completing tasks on your site.

In robots.txt: User-agent: Manus-User #

Fetches a page when a person asks Meta AI for it, or when an AI agent carries out a task for someone. Meta says it may not follow robots.txt for these requests.

Block it? It opens pages when someone asks Meta AI about them. Meta says it may not follow robots.txt for these requests.

In robots.txt: User-agent: Meta-ExternalFetcher #

Indexes pages to improve the search results behind Meta AI.

Block it? It helps AI tools find pages to cite. Blocking it makes Meta's answers less likely to include or link to you.

In robots.txt: User-agent: Meta-WebIndexer #

Collects pages to build datasets for training Mistral's AI models. Mistral says it isn't used for search or to answer live questions.

Block it? An AI company's bot. Block it if you don't want Mistral AI using your content.

In robots.txt: User-agent: MistralAI-Training #

Fetches pages when someone using Mistral's Le Chat assistant asks for them.

Block it? It opens a page when someone asks Mistral AI's assistant about it. Blocking it stops that, which can cost you citations.

In robots.txt: User-agent: MistralAI-User #

Listed by bot directories as collecting pages to train AI models. Moonshot's own crawler page names only KimiBot, Kimi-User and Kimi-SearchBot.

Block it? An AI company's bot. Block it if you don't want Moonshot AI using your content.

In robots.txt: User-agent: MoonshotBot #

Fetches pages for Tabstack, Mozilla's service that runs web browsing for other companies' AI agents.

Block it? An AI agent acting for a person, such as booking or shopping. Blocking it stops those users completing tasks on your site.

In robots.txt: User-agent: Mozilla-Tabstack #

Amazon Nova Act, an AI agent that uses a web browser to carry out tasks. Developers run it, and Amazon says the name appears in its default browser settings.

Block it? An AI agent acting for a person, such as booking or shopping. Blocking it stops those users completing tasks on your site.

In robots.txt: User-agent: NovaAct #

An older Webz.io crawler. Webz.io sells the web data it collects to other companies, including for AI training.

Block it? An AI company's bot. Block it if you don't want Webz.io using your content.

In robots.txt: User-agent: omgili #

An older Webz.io crawler. Webz.io sells the web data it collects to other companies, including for AI training.

Block it? An AI company's bot. Block it if you don't want Webz.io using your content.

In robots.txt: User-agent: Omgilibot #

Collects pages to train Huawei's PanGu AI models.

Block it? An AI company's bot. Block it if you don't want Huawei using your content.

In robots.txt: User-agent: PanguBot #

Fetches pages for Poggio, an AI sales tool, to collect citations and supporting information for its AI-written answers.

Block it? An AI company's bot. Block it if you don't want Poggio using your content.

In robots.txt: User-agent: Poggio-Citations #

Collects pages for an AI research lab. The bare token `poseidon` is not documented by anyone.

Block it? An AI company's bot. Block it if you don't want Poseidon Research using your content.

In robots.txt: User-agent: Poseidon Research Crawler #

Collects pages for SB Intuitions, which says it uses them for AI development and information analysis. It can be blocked in robots.txt.

Block it? An AI company's bot. Block it if you don't want SB Intuitions using your content.

In robots.txt: User-agent: SBIntuitionsBot #

An older name for ScalePostBot, which fetches individual pages when a ScalePost customer asks for them to be analysed. The bot now identifies as ScalePostBot, so add a rule for that name too.

Block it? An old name. The bot now identifies as ScalePostBot.

In robots.txt: User-agent: ScalePostAI #

Fetches individual pages when a ScalePost customer asks for them to be analysed. It doesn't crawl whole sites.

Block it? An AI company's bot. Block it if you don't want ScalePost using your content.

In robots.txt: User-agent: ScalePostBot #

Listed by bot directories as crawling sites to supply content to AI systems as a third-party service. Scry's own site doesn't mention it.

Block it? An AI company's bot. Block it if you don't want Scry using your content.

In robots.txt: User-agent: ScryBot #

Linked to Seekr, an AI company. We couldn't confirm exactly what it collects pages for.

Block it? An AI company's bot. Block it if you don't want Seekr using your content.

In robots.txt: User-agent: Seekr #

Linked to Seekr, an AI company. We couldn't confirm exactly what it collects pages for.

Block it? An AI company's bot. Block it if you don't want Seekr using your content.

In robots.txt: User-agent: SeekrBot #

Indexes pages for Parallel's search and extraction APIs, which AI agents query.

Block it? It helps AI tools find pages to cite. Blocking it makes Parallel's answers less likely to include or link to you.

In robots.txt: User-agent: ShapBot #

Tells Webz.io whether web data it collects can be used for AI training. Webz.io sells web data to other companies.

Block it? An AI company's bot. Block it if you don't want Webz.io using your content.

In robots.txt: User-agent: Webzio-Extended #

Listed by bot directories as an xAI crawler linked to Grok. xAI publishes nothing saying what it collects pages for.

Block it? An AI company's bot. Block it if you don't want xAI using your content.

In robots.txt: User-agent: xAI-Grok #

Listed by bot directories as collecting pages to train 01.AI's Yi models. 01.AI publishes nothing confirming it.

Block it? An AI company's bot. Block it if you don't want 01.AI using your content.

In robots.txt: User-agent: YiBot #

Checks the quality of pages used in Google Ads campaigns.

Block it? Google Ads uses it to check landing pages. If you advertise on Google, blocking it can hurt your campaigns.

In robots.txt: User-agent: AdsBot-Google #

Checks the quality of mobile pages used in Google Ads campaigns.

Block it? Google Ads uses it to check landing pages. If you advertise on Google, blocking it can hurt your campaigns.

In robots.txt: User-agent: AdsBot-Google-Mobile #

A retired Google Ads crawler that checked the quality of pages used in Android app ads. Google says it is no longer in use, so the rule does nothing now.

Block it? Google has retired this crawler, so the rule does nothing.

In robots.txt: User-agent: AdsBot-Google-Mobile-Apps #

Fetches feeds, pages and images for Apple News, so publisher channels display properly.

Block it? Apple News uses it to display your channel and articles. Blocking it can break them.

In robots.txt: User-agent: AppleNewsBot #

Collects metadata from page headers to support Audigent's targeted advertising products.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Audigent, blocking it can lower your ad revenue.

In robots.txt: User-agent: AudigentAdBot #

Fetches the headline and image when a link is shared on Facebook and other Meta apps. Blocking it can break those previews. Meta says it may ignore robots.txt when checking a link for malware or other security problems.

Block it? Blocking it breaks the headline and image preview when people share your links on Meta.

In robots.txt: User-agent: facebookexternalhit #

Fetches the headline and image when a link is shared on LinkedIn. Blocking it can break those previews.

Block it? Blocking it breaks the headline and image preview when people share your links on LinkedIn.

In robots.txt: User-agent: LinkedInBot #

Reads a page's meta tags to list it in the Livelap discovery app. Its own documentation says it doesn't read robots.txt.

Block it? Livelap says it doesn't read robots.txt.

In robots.txt: User-agent: LivelapBot #

Reads pages for Mantis so advertisers can match ads to page content and avoid unsuitable pages.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Mantis, blocking it can lower your ad revenue.

In robots.txt: User-agent: Mantisbot #

Reads pages for Google AdSense, so relevant ads can be shown. Blocking it can reduce ad revenue on AdSense pages.

Block it? AdSense uses it to choose ads. If you run AdSense, blocking it can cut your ad revenue.

In robots.txt: User-agent: Mediapartners-Google #

Crawls pages to improve Meta's advertising and other business products.

Block it? It helps Meta's advertising products. Blocking it is low risk unless you advertise with Meta.

In robots.txt: User-agent: Meta-ExternalAds #

Checks the safety of pages submitted as ads on ChatGPT. It only visits those pages, and OpenAI says the data isn't used to train its models.

Block it? It only matters if you advertise on ChatGPT. Blocking it may stop your ads being approved.

In robots.txt: User-agent: OAI-AdsBot #

Reads pages on publisher sites to work out their topics, which Outbrain uses for content recommendations and advertising.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Outbrain, blocking it can lower your ad revenue.

In robots.txt: User-agent: outbrain #

Matches one of the names Pinterest's crawler uses. Pinterest tells site owners to use the name Pinterestbot in robots.txt, so this rule may not match.

Block it? Pinterest's crawler uses the name Pinterestbot, so this rule may not match.

In robots.txt: User-agent: Pinterest #

Crawls pages and images for Pinterest.

Block it? Blocking it keeps your pages and images off Pinterest.

In robots.txt: User-agent: Pinterestbot #

Fetches a page when someone posts its link on Reddit, to build the preview.

Block it? Blocking it breaks the headline and image preview when people share your links on Reddit.

In robots.txt: User-agent: RedditBot #

Reads page content so Sirdata can categorise it for advertisers targeting by context.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Sirdata, blocking it can lower your ad revenue.

In robots.txt: User-agent: SirdataBot #

Fetches link previews when a link is shared in Slack.

Block it? Blocking it breaks the headline and image preview when people share your links on Slack.

In robots.txt: User-agent: Slackbot #

Reads page content so Smartology can match adverts to the subject of the page.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Smartology, blocking it can lower your ad revenue.

In robots.txt: User-agent: SmartologyBot #

Fetches a page when someone shares its link on TikTok, to build the preview card.

Block it? Blocking it breaks the headline and image preview when people share your links on ByteDance.

In robots.txt: User-agent: TikTokSpider #

Scans pages that request adverts through The Trade Desk, so advertisers can place ads next to content they consider relevant.

Block it? Advertisers use it to match ads to your pages. If you sell ads through The Trade Desk, blocking it can lower your ad revenue.

In robots.txt: User-agent: TTD-Content #

Fetches the headline and image when a link is shared on X. Blocking it can break those previews.

Block it? Blocking it breaks the headline and image preview when people share your links on X.

In robots.txt: User-agent: Twitterbot #

Crawls websites that use AddSearch, a site search product, so their own search boxes can return results.

Block it? It only crawls sites that AddSearch's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: AddSearchBot #

Reads page text to classify it for contextual advertising. Older branding that IAS no longer documents.

Block it? An old or unused name, so a rule for it probably does nothing. Use the bot's current name instead.

In robots.txt: User-agent: ADmantX #

Collects company details, team members and the technology each site uses, for AgentData's company data product aimed at AI agents and sales teams.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: AgentDataBot #

Crawls the web for Ahrefs, an SEO tool, and for Yep, Ahrefs' search engine. Blocking it makes Ahrefs' data about your site less complete and keeps your pages out of Yep. It doesn't affect Google.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from Ahrefs' data, which your competitors can also read.

In robots.txt: User-agent: AhrefsBot #

Used when someone runs an Ahrefs Site Audit on your site. It follows robots.txt unless a verified owner of the site turns that off for their own audits. Blocking it may stop your own team's audits working.

Block it? It runs when someone audits your site with Ahrefs. Keep it if your team or agency uses Ahrefs.

In robots.txt: User-agent: AhrefsSiteAudit #

Collects information about companies from their websites for aiHit, a business data provider.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: aiHitBot #

Scans sites for security problems as part of Alibaba Cloud's security services.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: AliyunSecBot #

Classifies pages that show Amazon ads, so ads suit the content.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Amazon Ads, blocking it can lower your ad revenue.

In robots.txt: User-agent: AmazonAdBot #

Scrapes page content for Apify customers, often to feed AI applications.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Apify Website Content Crawler #

Scrapes pages for Apify customers, often feeding AI and automation workflows.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: ApifyBot #

Used by Awario, a tool that tracks where brands are mentioned online.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: AwarioRssBot #

Used by Awario, a tool that tracks where brands are mentioned online.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: AwarioSmartBot #

Crawls the web to map the links between sites, which feeds Babbar's SEO tools.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from Babbar's data, which your competitors can also read.

In robots.txt: User-agent: Barkrowler #

Amazon Bedrock's crawler, which an AWS customer points at pages they're allowed to crawl, to build their own knowledge base.

Block it? It only crawls sites that Amazon (AWS)'s customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: Bedrockbot #

Reads companies' public pages to build Bixel's company profiles, used by investors and analysts.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: BixelBot #

Builds the backlink index behind SEO PowerSuite's link tools.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from SEO PowerSuite's data, which your competitors can also read.

In robots.txt: User-agent: BLEXBot #

The crawler for Botify, an SEO platform that crawls sites for its customers. Botify says that without a rule of its own it follows Googlebot's rules, then the rules for all bots.

Block it? It only crawls sites that Botify's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: Botify #

Aimed at Brandwatch, a social media monitoring company. Its crawler identifies itself as magpie-crawler, so a rule for this name may not match.

Block it? Brandwatch's crawler identifies as magpie-crawler, so this rule may not match.

In robots.txt: User-agent: Brandwatch #

Collects pages for Bright Data's commercial web data products and datasets.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Brightbot #

Bright Data's crawler. Bright Data says it is the main data collection system behind its web data products.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Brightbot 1.0 #

Crawls customer sites for BrightEdge's enterprise SEO platform.

Block it? It only crawls sites that BrightEdge's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: BrightEdge #

Monitors news sites for Hypefactors, a media monitoring platform used by PR teams.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Buck #

Crawls sites for Cloudflare customers using the crawl feature of Browser Run (formerly Browser Rendering), so the purpose depends on the customer. Cloudflare says it follows robots.txt, including Crawl-delay and Content Signals.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: CloudflareBrowserRenderingCrawler #

Runs crawlers that other developers build on the Crawlspace platform, so the purpose varies by customer.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Crawlspace #

Classifies what a page is about so Criteo can match ads to it.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Criteo, blocking it can lower your ad revenue.

In robots.txt: User-agent: CriteoBot #

Crawls sites for the backlink database DataForSEO sells to SEO tools.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from DataForSEO's data, which your competitors can also read.

In robots.txt: User-agent: DataForSeoBot #

Reported to be a crawler from netEstate, a German company that collects and sells website data.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Datenbank Crawler #

The crawler for Lumar, a website auditing platform that used to be called Deepcrawl. Lumar says a rule for either name blocks it.

Block it? It only crawls sites that Lumar's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: Deepcrawl #

Builds Moz's database of links between websites, used by SEO teams.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from Moz's data, which your competitors can also read.

In robots.txt: User-agent: DotBot #

Fetches publishers' articles for Echobox's social and newsletter automation, usually under contract to that publisher.

Block it? Publishers use Echobox to post their articles to social media and newsletters. Blocking it can break that.

In robots.txt: User-agent: EchoboxBot #

Reported to gather financial and business data from websites for FactSet, a financial data company.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: FactSet_spyderbot #

Fetches pages and converts them to text for customers building AI applications.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: FirecrawlAgent #

Collects news, government, regulatory and industry content for Gnowit's media and policy monitoring service.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: GnowitNewsbot #

Read page text for contextual advertising. Oracle closed its advertising business in September 2024, so this rule no longer does anything.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: Grapeshot #

Rates page content for brand safety, which decides whether advertisers buy space on it.

Block it? Advertisers use its brand-safety ratings to decide whether to buy ads on your pages. Publishers who sell ads should keep it.

In robots.txt: User-agent: IAS Crawler #

Scans sites to score an organisation's security posture. Reported not to follow robots.txt.

Block it? Reported not to follow robots.txt, so a rule here may have no effect.

In robots.txt: User-agent: ISSCyberRiskCrawler #

Reported to collect text from websites for the University of Leipzig's language research collections.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: LCC #

A crawler from Linkfluence, now part of Meltwater, which gathers web content for its monitoring products.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Linkfluence #

Collects blogs, forums and news so Brandwatch customers can track brand mentions.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: magpie-crawler #

Collects news articles for an open academic archive used to study how news spreads.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: MediaCloud #

Fetches news and web pages for Meltwater's media monitoring products.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: Meltwater #

Builds Majestic's database of links between websites, used by SEO teams.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from Majestic's data, which your competitors can also read.

In robots.txt: User-agent: MJ12bot #

A link checker. Schools and colleges using Moodle run it to find broken links in their courses.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: MoodleBot #

Finds a website's imprint (legal notice) page and extracts addresses, contact details and company names. netEstate says it respects robots.txt.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: netEstate Imprint Crawler #

Extracts business and contact details, including German imprint pages, for commercial data products.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: netEstate NE Crawler #

Crawls websites to extract public contact and business information for netEstate, a German data company.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: netEstate NE Crawler #

Fetched feeds and public pages to show in users' Netvibes dashboards. The standalone Netvibes service closed in June 2025.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: Netvibes #

Crawls a site when its owner runs a technical SEO audit.

Block it? It only crawls sites that Oncrawl's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: Oncrawl #

Crawls public sites and turns them into licensed datasets about businesses and professionals.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: panscient.com #

Crawls public sites and turns them into licensed datasets about businesses and professionals.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: panscient.com #

Reads page content so advertisers can target or avoid it without using cookies.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Peer39, blocking it can lower your ad revenue.

In robots.txt: User-agent: Peer39 Crawler #

Builds the index behind Pipl's people-search product.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: PiplBot #

Reads page text so Comscore can classify the content for advertisers targeting by context.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Comscore, blocking it can lower your ad revenue.

In robots.txt: User-agent: Proximic #

Reads a customer's own website so Qualified's sales chatbots can answer questions about it.

Block it? It only crawls sites that Qualified's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: QualifiedBot #

Builds audience categories from page content. Quantcast follows robots.txt when crawling, but not when checking an advert's landing page.

Block it? Advertisers use it to match ads to your pages. If you sell ads through Quantcast, blocking it can lower your ad revenue.

In robots.txt: User-agent: QuantcastBot #

Used when someone runs a Moz site crawl.

Block it? It runs when someone crawls your site with Moz. Keep it if your team or agency uses Moz.

In robots.txt: User-agent: rogerbot #

Crawls the web for Semrush, an SEO tool. Blocking it makes Semrush's data about your site less complete. It doesn't affect Google.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from Semrush's data, which your competitors can also read.

In robots.txt: User-agent: SemrushBot #

Crawls for Semrush's AI content tool. Often listed as an AI training crawler; on Semrush's own description it isn't one.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: SemrushBot-OCOB #

Checks that URLs work, for Semrush's SEO Writing Assistant.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: SemrushBot-SWA #

Collects comments, posts and other user-written content for SentiOne's social listening platform. SentiOne says it follows rules for sentibot, or Googlebot's rules if there are none.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: SentiBot #

Used by SentiOne, a tool that tracks where brands are mentioned online.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: SentiOne #

Records the links a site makes to other domains, for the German backlink tool SEOkicks.

Block it? It doesn't send you visitors. Blocking it saves server load and hides your site from SEOkicks' data, which your competitors can also read.

In robots.txt: User-agent: SEOkicks #

An older name for the SEOkicks backlink crawler, which now identifies itself as SEOkicks.

Block it? An old name. The crawler now identifies as SEOkicks.

In robots.txt: User-agent: SEOkicks-Robot #

Semrush's site audit crawler. It checks a site for SEO and technical problems when someone runs a Semrush audit on it.

Block it? It runs when someone audits your site with Semrush. Keep it if your team or agency uses Semrush.

In robots.txt: User-agent: SiteAuditBot #

Builds Tavily's web index, which grounds AI agents in current page content.

Block it? It helps AI tools find pages to cite. Blocking it makes Tavily's answers less likely to include or link to you.

In robots.txt: User-agent: TavilyBot #

Crawls public news sites, forums and blogs, including comments, for media monitoring. Trendiction says the data goes to clients such as market research and marketing firms.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: trendictionbot #

Collects web pages for Turnitin's plagiarism checker, which compares student work against content on the internet.

Block it? It lets Turnitin check student work against your pages. Blocking it doesn't affect visitors.

In robots.txt: User-agent: Turnitin #

Collects pages so Turnitin can check student work for copied text.

Block it? It lets Turnitin check student work against your pages. Blocking it doesn't affect visitors.

In robots.txt: User-agent: TurnitinBot #

Checks whether a website is up and how fast it responds, for site owners who use the UptimeRobot monitoring service.

Block it? It checks whether your site is up. If you use UptimeRobot, blocking it can trigger false outage alerts.

In robots.txt: User-agent: UptimeRobot #

Scans public pages to build business contact datasets for Hunter's B2B products.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: VelenPublicWebCrawler #

A website monitoring crawler. It checks page status codes to see whether pages are available.

Block it? It doesn't send you visitors. Blocking it saves server load, but check it isn't a tool you or your agency use.

In robots.txt: User-agent: WARDBot #

Linked to Archive-It, the Internet Archive's service for libraries and universities that build their own collections. Its crawler identifies as archive.org_bot, so a rule for this name may not apply to it.

Block it? Archive-It's crawler identifies as archive.org_bot, so this rule may not apply. Use archive.org_bot.

In robots.txt: User-agent: archive-it.org #

Saves copies of pages for the Wayback Machine and for Archive-It collections.

Block it? Blocking it keeps future copies of your pages out of Internet Archive's archive. It doesn't affect visitors.

In robots.txt: User-agent: archive.org_bot #

A volunteer-run tool from Archive Team that saves copies of websites, which usually end up in the Wayback Machine. Archive Team says it doesn't act on robots.txt rules, so this rule does nothing.

Block it? The operator says it doesn't follow robots.txt, so a rule here has little or no effect.

In robots.txt: User-agent: ArchiveBot #

Saves copies of websites for Arquivo.pt, Portugal's national web archive.

Block it? Blocking it keeps future copies of your pages out of Arquivo.pt's archive. It doesn't affect visitors.

In robots.txt: User-agent: arquivo-web-crawler #

Copies a writer's published work for them, so it survives if the original site changes or disappears.

Block it? Writers use it to keep copies of their own published work. Blocking it stops your writers backing up their articles.

In robots.txt: User-agent: Authory #

Archives UK websites for the British Library and the other legal deposit libraries. They say it generally follows robots.txt but may overrule it to capture a page properly.

Block it? It archives UK websites for the legal deposit libraries. They may overrule the block, as the law lets them.

In robots.txt: User-agent: bl.uk_lddc_bot #

Crawled European websites for preservation. The organisation closed in 2018.

Block it? The service has closed, so a rule for it does nothing. It's safe to remove.

In robots.txt: User-agent: europarchive.org #

An older name linked to the Wayback Machine and the now-closed Alexa rankings.

Block it? Blocking it keeps future copies of your pages out of Internet Archive's archive. It doesn't affect visitors.

In robots.txt: User-agent: ia_archiver #

Harvests Swedish websites for the national web archive.

Block it? Blocking it keeps future copies of your pages out of National Library of Sweden's archive. It doesn't affect visitors.

In robots.txt: User-agent: Kulturarw3 #

Captures whole websites for compliance archives, including the UK Government Web Archive.

Block it? It captures websites for compliance archives, including the UK Government Web Archive.

In robots.txt: User-agent: MirrorWeb #

Harvests Danish websites for Denmark's legal-deposit archive. The library states it doesn't follow robots.txt. Its crawler now identifies as heritrix with a kb.dk link, so this name is unlikely to match it anyway.

Block it? The operator says it doesn't follow robots.txt, so a rule here has little or no effect.

In robots.txt: User-agent: netarkivet.dk #

Archives a customer's own website on a schedule, for compliance and records requests.

Block it? It only crawls sites that Pagefreezer's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: PageFreezer #

Captures pages to build a tamper-proof record for regulatory compliance.

Block it? It only crawls sites that Smarsh's customers connect. Fine to block unless your organisation uses it.

In robots.txt: User-agent: SmarshBot #

Harvests Czech websites for the national web archive.

Block it? Blocking it keeps future copies of your pages out of National Library of the Czech Republic's archive. It doesn't affect visitors.

In robots.txt: User-agent: webarchiv.cz #

The name used by Lighthouse, Google's open-source page auditing tool, which anyone can run.

Block it? Google's page-testing tool. If you block it, page speed tests on your own site may fail.

In robots.txt: User-agent: Chrome-Lighthouse #

The default name used by crawler4j, an open-source web crawler library for Java. Anyone can run it, for any purpose.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: crawler4j #

Open-source archiving software, used by the Internet Archive and by other libraries and companies.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Heritrix #

Free software that copies a whole website to a local folder for offline browsing. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: HTTrack #

An open-source tool anyone can run to turn lists of image links into AI training datasets. The name says what software is running, not who is running it.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: img2dataset #

Old open-source crawling software, built to fetch large numbers of pages for a search engine. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: larbin #

An early web software library from the W3C. Anyone can build a program with it, so the name says what software is fetching, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: libwww #

A free headless browser that anyone can run, often for AI agents and scraping. It only follows robots.txt if the person running it switches that on.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Lightpanda #

The default name of an open-source news crawler that anyone can run to extract articles.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: news-please #

Open-source crawling software anyone can run. The name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Nutch #

A Windows program that downloads websites to a hard disk for offline browsing. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Offline Explorer #

The default name used by Scrapy, an open-source scraping framework. Anyone can run it, for any purpose.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Scrapy #

Desktop software that SEO teams run to audit websites. The name says which program is crawling, not who is running it.

Block it? SEO teams run it to audit sites. Keep it if your team or agency uses Screaming Frog.

In robots.txt: User-agent: Screaming Frog SEO Spider #

Matches Teleport Pro, an old Windows program that downloads whole websites for offline browsing. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Teleport #

Teleport Pro, an old Windows program that downloads whole websites for offline browsing. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: TeleportPro #

A program that copies websites to a hard disk for offline browsing. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: WebCopier #

An offline browser that downloads web pages or whole sites to a hard disk. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: WebZIP #

A free command-line tool for downloading files from the web. Anyone can run it, so the name says what software is fetching, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Wget #

Matches Xenu's Link Sleuth, a free Windows program that checks websites for broken links. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Xenu #

A free Windows program that checks websites for broken links. Anyone can run it, so the name says what software is crawling, not who.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: Xenu Link Sleuth #

The crawler name used by YaCy, free search software that anyone can run, so these requests can come from many different people.

Block it? Anyone can run this software, often to copy or scrape a site. Blocking it is low risk, but a scraper can simply change its name.

In robots.txt: User-agent: yacybot #