Agent Directory
The definitive guide to every known AI agent, crawler, scraper, and other bot on the internet, updated daily.
AI Data Scrapers
MistralBot
MistralBot is a web crawler operated by Mistral, a company that develops frontier language models and enterprise AI platforms. The bot collects data from web pages to sup…
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
MoonshotBot
MoonshotBot is a web crawler operated by Moonshot AI that downloads website content for Kimi and related Moonshot AI services.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
MoonSpider
MoonSpider is a web crawler operated by Moonshot AI that downloads website content for AI model training and research.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
netEstate Imprint Crawler
netEstate Imprint Crawler is an AI data scraper operated by netEstate.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
omgili
omgili is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
PanguBot
PanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
QwenBot
QwenBot is a web crawler operated by Alibaba that downloads website content to support Qwen training and generated answers across Alibaba services.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
recraft-image-dataset
recraft-image-dataset collects publicly accessible images for datasets used to train and improve Recraft's image-generation models and design platform.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
ReflectionBot
Reflectionbot is a web crawler operated by Reflection that collects website content to train its open AI models.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
SalamandraVLM
SalamandraVLM is a web crawler operated by Aithlas that downloads website content for AI vision-language model research.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
SBIntuitionsBot
SBIntuitionsBot is a web crawler operated by SB Intuitions, a Japanese company that develops generative language models optimized for the Japanese language and culture. T…
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
Spider
Spider is a web crawler designed for AI projects, including AI agents, LLMs, RAG systems, and data analysis. It collects and converts web data into multiple formats inclu…
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
Timpibot
Timpibot is used by Timpi's decentralized network of independent node operators. The index they build can be used to train LLMs (Large Language Models).
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
VelenPublicWebCrawler
VelenPublicWebCrawler is a web crawler developed by Velen for Hunter that analyzes millions of publicly accessible internet pages every month. The bot builds business dat…
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
webzio-extended
webzio-extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
YiBot
YiBot crawls public web content for 01.AI, the company behind the Yi family of foundation models and enterprise AI products.
AI Data Scraper
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
AI Search Crawlers
AddSearchBot
AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
AIWebIndex
AIWebIndex is Lyrenth's scheduled crawler for building an AI-readable index of the public web. Pages are stored and served to AI agents as clean, attributed documents.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
AIWebIndex-Agent
AIWebIndex-Agent fetches a page into Lyrenth's index when an agent requests a URL that is not yet indexed or explicitly requests a fresh copy.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
amazon-kendra
amazon-kendra is an Amazon web crawler that indexes authorized web pages for Amazon Kendra search applications.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Amzn-SearchBot
Amzn-SearchBot is an AI search crawler operated by Amazon that indexes web content for use in improving Alexa and other Amazon services.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
AndiAgent
AndiAgent is the web crawler for Andi and indexes public web content for its AI search engine.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Anomura
Anomura is Direqt's web crawler that discovers and indexes links and metadata from websites for inclusion in Direqt's AI search results.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Applebot
Applebot is Apple's web crawler that gathers data to power search features across Apple's ecosystem including Spotlight, Siri, and Safari search functionality.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
atlassian-bot
atlassian-bot is Atlassian's AI web crawler that indexes custom websites for Rovo, allowing the indexed content to appear in Rovo Search results and be used by Rovo Chat …
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
AzureAI-SearchBot
AzureAI-SearchBot is an AI search crawler operated by Microsoft that indexes web content to support Azure AI services and improve search capabilities across Microsoft's A…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Channel3Bot
Channel3Bot is Channel3's web crawler that visits publicly accessible product detail pages to index product information into a universal product catalog. The bot collects…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Claude-SearchBot
Claude-SearchBot is used to create an index of websites that can be surfaced as results in Anthropic's Claude AI assistant search feature.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Cloudflare-AutoRAG
Cloudflare-AutoRAG is Cloudflare's web crawler that indexes websites for AI Search (formerly AutoRAG), a managed retrieval-augmented generation service. The crawler autom…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Dola-SearchBot
Dola-SearchBot is a web crawler operated by ByteDance that appears to collect web content for search features in its Dola AI assistant.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
ExaSearchBot
ExaSearchBot is Exa's crawler for discovering and indexing public web pages so users can find, retrieve, and cite their content through Exa.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Google-CloudVertexBot
Google-CloudVertexBot is a web crawler operated by Google that collects content for Google Cloud's Vertex AI Search service. This crawler indexes web pages to power enter…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
IsonSearchBot
IsonSearchBot crawls Indonesian-language and Indonesia-focused web pages to build the index behind Ison Search. The index supports search results and cited AI answers tha…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Kimi-SearchBot
Kimi-SearchBot is a web crawler operated by Moonshot AI that powers Kimi's search functionality. It analyzes and indexes web pages so they can appear in Kimi search resul…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
LinkupBot
LinkupBot is a web crawler operated by Linkup that indexes enterprise web content for its AI search platform. The bot visits websites to collect relevant business informa…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
meta-webindexer
meta-webindexer is Meta's web crawler that browses the internet to improve search results for Meta AI users. It analyzes online content to make Meta AI's responses more r…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
MistralAI-Index
MistralAI-Index is one of Mistral AI's automated web crawlers. It indexes web content to power search capabilities in Vibe, Mistral's AI assistant product.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
OAI-SearchBot
OAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time informatio…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
PerplexityBot
PerplexityBot is the web crawler for Perplexity AI that indexes web content to power the AI search engine's real-time information retrieval and answer generation capabili…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
PetalBot
PetalBot is Huawei's web crawler that indexes PC and mobile websites to build search databases for Petal Search engine and provide AI-powered content recommendations for …
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
TalarionSearchBot
TalarionSearchBot is a web crawler operated by Talarion that indexes recent web content for its AI search API and knowledge base.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
xAI-SearchBot
xAI-SearchBot is a web crawler operated by xAI, the company behind Grok, to collect information from across the web for use in their search and chatbot products. This bot…
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
YandexAdditional
YandexAdditional is a web crawler operated by Yandex that retrieves website content for YandexGPT quick answers in Yandex Search.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
YandexAdditionalBot
YandexAdditionalBot is a web crawler operated by Yandex that retrieves website content for YandexGPT quick answers in Yandex Search.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
YepBot
YepBot is a web crawler operated by Ahrefs that supports Yep, a search engine providing web search results for human users and AI agents.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
ZanistaBot
ZanistaBot is a web crawler operated by Zanista that indexes public web content for its search platform.
AI Search Crawler
AI Search Crawler
Indexes website content to possibly include as citations in AI-powered search results
See More →
Archivers
Archive-It
Archive-It is a web archiving crawler operated by Internet Archive that preserves copies of web pages for long-term digital preservation and historical record-keeping. It…
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
See More →
archive.org_bot
archive.org_bot is the Internet Archive's web crawler for the Wayback Machine, systematically crawling and preserving publicly accessible web pages for historical record …
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
See More →
Arquivo-web-crawler
Arquivo-web-crawler is the Portuguese web archive's bot that systematically crawls and preserves Portuguese websites for historical research, creating a comprehensive dig…
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
See More →
Authory
Authory is an automated content archiving crawler that systematically searches for and backs up published articles, podcasts, and videos by journalists and content creato…
Archiver
Archiver
Captures and stores historical website snapshots for long-term digital preservation
See More →