Agents
AI Search Crawlers
Every known artificial agent (bot) on the internet. You can track their activity on your website with Agent Analytics, or control their behavior with Automatic Robots.txt.
AI Search Crawlers
AddSearchBot
AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results.
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Amzn-SearchBot
Amzn-SearchBot is an AI search crawler operated by Amazon that indexes web content for use in improving Alexa and other Amazon services.
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Anomura
Anomura is Direqt's web crawler that discovers and indexes links and metadata from websites for inclusion in Direqt's AI search results.
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Applebot
Applebot is Apple's web crawler that gathers data to power search features across Apple's ecosystem including Spotlight, Siri, and Safari search functionality.
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
atlassian-bot
atlassian-bot is Atlassian's AI web crawler that indexes custom websites for Rovo, allowing the indexed content to appear in Rovo Search results and be used by Rovo Chat …
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
AzureAI-SearchBot
AzureAI-SearchBot is an AI search crawler operated by Microsoft that indexes web content to support Azure AI services and improve search capabilities across Microsoft's A…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Channel3Bot
Channel3Bot is Channel3's web crawler that visits publicly accessible product detail pages to index product information into a universal product catalog. The bot collects…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Claude-SearchBot
Claude-SearchBot is used to create an index of websites that can be surfaced as results in Anthropic's Claude AI assistant search feature.
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Cloudflare-AutoRAG
Cloudflare-AutoRAG is Cloudflare's web crawler that indexes websites for AI Search (formerly AutoRAG), a managed retrieval-augmented generation service. The crawler autom…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Google-CloudVertexBot
Google-CloudVertexBot is a web crawler operated by Google that collects content for Google Cloud's Vertex AI Search service. This crawler indexes web pages to power enter…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
LinkupBot
LinkupBot is a web crawler operated by Linkup that indexes enterprise web content for its AI search platform. The bot visits websites to collect relevant business informa…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
meta-webindexer
meta-webindexer is Meta's web crawler that browses the internet to improve search results for Meta AI users. It analyzes online content to make Meta AI's responses more r…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
OAI-SearchBot
OAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time informatio…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
PerplexityBot
PerplexityBot is the web crawler for Perplexity AI that indexes web content to power the AI search engine's real-time information retrieval and answer generation capabili…
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
PetalBot
PetalBot is Huawei's web crawler that indexes PC and mobile websites to build search databases for Petal Search engine and provide AI-powered content recommendations for …
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
ZanistaBot
ZanistaBot is an AI search crawler operated by Zanista. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Page 1 of 1