Agents
All Agent Types
Every known artificial agent (bot) on the internet. You can track their activity on your website with Agent Analytics, or control their behavior with Automatic Robots.txt.
AI Agents
AmazonBuyForMe
AmazonBuyForMe is an Amazon bot that crawls websites as part of the Amazon Buy For Me service. This bot visits product pages and e-commerce websites to gather product inf…
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
See More →
ChatGPT Agent
ChatGPT Agent is an AI agent created by OpenAI that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf…
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
See More →
Google-Agent
Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
See More →
GoogleAgent-Mariner
GoogleAgent-Mariner is an AI agent created by Google that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on …
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
See More →
Manus-User
Manus-User is a browser-enabled AI agent operated by Butterfly Effect, a company based in China. It autonomously navigates websites, interprets content, and carries out m…
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
See More →
NovaAct
Nova Act is an AI agent created by Amazon that can use a web browser. It can intelligently navigate and interact with websites to complete multi-step tasks on behalf of a…
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
See More →
TwinAgent
TwinAgent is operated by Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa…
AI Agent
Uses an actual web browser to autonomously complete complex tasks on behalf of a human user
See More →
AI Assistants
AI2Bot-DeepResearchEval
Ai2Bot-DeepResearchEval is operated by Ai2, a non-profit AI research institute. It's used to collect and scan resources used in deep research queries performed by Ai2's o…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
Amzn-User
Amzn-User is an AI assistant operated by Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services.
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
bigsur.ai
bigsur.ai is a web crawler operated by Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
ChatGPT-User
ChatGPT-User is OpenAI's web crawler that visits websites when ChatGPT users request information. This enables ChatGPT to include links in its responses.
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
Claude-User
Claude-User is dispatched by Anthropic's Claude AI assistant in response to user prompts, when it needs to fetch content to include in its answers.
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
DuckAssistBot
DuckAssistBot is a web crawler that scans websites to collect content for DuckDuckGo's AI-assisted answers feature, which generates brief responses to search queries usin…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
Gemini-Deep-Research
Gemini-Deep-Research is the agent responsible for collecting and scanning resources used in Google Gemini's Deep Research feature, which acts as a personal research assis…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
Google-NotebookLM
Google-NotebookLM is an AI-powered research and note-taking assistant that helps users synthesize information from uploaded sources like documents, transcripts, or web co…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
kagi-fetcher
kagi-fetcher is an AI Assistant operated by Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
KlaviyoAIBot
KlaviyoAIBot is Klaviyo's web crawler that fetches publicly available pages from domains explicitly connected to user accounts to power the Kai Customer Agent feature. Th…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
LinerBot
LinerBot is the web crawler used by Liner AI assistant to gather information from academic sources and websites to provide accurate answers with line-by-line source citat…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
meta-externalfetcher
meta-externalfetcher is used by Meta to perform user-initiated fetches of individual links from AI assistant product functions.
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
MistralAI-User
MistralAI-User is Mistral's AI assistant bot that performs web browsing and data gathering tasks for users in Le Chat, including opening web pages and retrieving informat…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
Perplexity-User
Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a …
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
PhindBot
Phind is an AI-powered answer engine designed for developers, offering technical answers and code examples. It uses real-time web search and specialized AI models to prov…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
Poggio-Citations
Poggio-Citations is a web crawler operated by Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
QualifiedBot
QualifiedBot is Qualified's web crawler that analyzes customer websites to provide contextual information for their AI-powered chatbots and conversational marketing platf…
AI Assistant
Fetches website content in response to a user prompt, to include in an AI-generated answer
See More →
AI Coding Agents
Claude-Code
Claude Code is an AI coding agent by Anthropic that can build, debug, and ship code directly from the terminal, handling tasks like codebase onboarding, multi-file edits,…
AI Coding Agent
Fetches documentation and other resources to help build software
See More →
Code
Code (GitHub Copilot) is an AI coding agent that can autonomously plan, build, and execute development tasks, functioning as a collaborative AI pair programmer.
AI Coding Agent
Fetches documentation and other resources to help build software
See More →
Devin
Devin is a software engineering AI assistant that can browse websites and perform web-based tasks, functioning as a collaborative AI teammate for engineering teams.
AI Coding Agent
Fetches documentation and other resources to help build software
See More →
Google-Gemini-CLI
Gemini CLI is an AI coding agent by Google that can query and edit large codebases, generate apps from images or PDFs, and automate complex workflows directly from the te…
AI Coding Agent
Fetches documentation and other resources to help build software
See More →
opencode
OpenCode is an open-source AI coding agent that helps developers write code from the terminal, IDE, or desktop, supporting multiple LLM providers and local models.
AI Coding Agent
Fetches documentation and other resources to help build software
See More →
Trae
Trae is an AI-powered coding agent developed by ByteDance that can understand codebases, fetch web content, and generate code.
AI Coding Agent
Fetches documentation and other resources to help build software
See More →
AI Data Providers
ApifyBot
ApifyBot is a web scraping and data extraction crawler by Apify that collects website content for use in AI, LLMs, RAG, and automation workflows.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
ApifyWebsiteContentCrawler
ApifyWebsiteContentCrawler is a web crawler by Apify that extracts and downloads full website content for use in AI, data analysis, and automation workflows.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
Bravebot
Bravebot is a web crawler by Brave that indexes pages for Brave Search, providing search data and AI-optimized context to power chatbots, agents, and RAG pipelines.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
Brightbot
Brightbot is a web data collection crawler by Bright Data that extracts and structures public website content at scale, providing AI-ready data for model training, RAG pi…
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
Diffbot
Diffbot is a web crawler that extracts and structures website content using AI-powered visual understanding, providing knowledge graph data for applications like market i…
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
ExaBot
ExaBot is a web crawler that indexes web content to power Exa's AI search engine and semantic search APIs for AI applications.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
FirecrawlAgent
FirecrawlAgent is a web crawler operated by Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
HenkBot
Henkbot crawls the web on behalf of Valyu, an AI search infrastructure provider that indexes content for use in AI-powered retrieval pipelines.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
ShapBot
ShapBot is a web crawler by Parallel that collects and structures web content to power its search, extraction, and deep research APIs, providing AI agents with high-accur…
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
TavilyBot
TavilyBot is a web crawler by Tavily that indexes and extracts content from billions of pages, providing real-time search, extraction, and research data to ground AI agen…
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
Terra Cotta
Terra Cotta is Ceramic's web crawler that indexes public content to power their web-scale search API for AI and LLMs.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
TerraCotta
TerraCotta is Ceramic's web crawler that indexes public content to power their web-scale search API for AI and LLMs.
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
YouBot
YouBot is a web crawler by You.com that indexes and extracts web content to power its real-time search, contents, and research APIs, delivering grounded web data to AI ag…
AI Data Provider
Crawls websites to supply structured content to AI systems as a third-party service
See More →
AI Data Scrapers
Ai2Bot-Dolma
Ai2Bot-Dolma is operated by Ai2, a non-profit AI research institute. It's used to download data to train open source AI models.
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
Amazonbot
Amazonbot is a web crawler operated by Amazon that builds an index of web content to improve Amazon products and services, including Alexa, Kindle, and Amazon Shopping. T…
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
Applebot-Extended
Apple-Extended is used to train Apple’s foundation LLM models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer…
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
Bytespider
Bytespider is a web crawler operated by ByteDance, the Chinese owner of TikTok. It's allegedly used to download training data for its LLMs (Large Language Model) includin…
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
CCBot
CCBot is Common Crawl's web crawler that creates an open repository of web data, making crawled content universally accessible for research, analysis, and AI training pur…
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
ChatGLM-Spider
ChatGLM-Spider is a web crawler operated by Zhipu AI, the Chinese company behind ChatGLM. It is used for collecting data to train and evaluate the company's large languag…
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
ClaudeBot
ClaudeBot is a web crawler operated by Anthropic to download training data for its LLMs (Large Language Models) that power AI products like Claude.
AI Data Scraper
Downloads website content to include in datasets used for training AI models such as LLMs
See More →
Page 1 of 34
Next →