Agents
All Agent Types
Every known artificial agent (bot) on the internet. You can track their activity on your website with Agent Analytics, or control their behavior with Automatic Robots.txt.
Intelligence Gatherers
slickstream
slickstream is an intelligence gatherer operated by Slickstream. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
SmartologyBot
SmartologyBot is a contextual advertising crawler that analyzes web content to generate semantic vectors for matching publisher content with relevant advertisements on th…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
SMTBot
SMTBot is a web crawler operated by Similarweb that scans websites to detect and catalog the technologies and software tools being used. The collected data powers Similar…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
StatistikAustria
StatistikAustria crawls e-commerce websites to collect product pricing data for calculating the official consumer price index of Austria, contributing to national economi…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Stripebot
Stripebot is a web crawler operated by Stripe, a payment processing platform, that fetches and analyzes web content related to merchant websites and online businesses tha…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Taboola
Taboola is an intelligence gatherer operated by Taboola. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Taboolabot
Taboolabot is a web crawler used by Taboola to audit advertiser campaign websites, collecting content metadata and site information to enable targeted content recommendat…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
trendictionbot
trendictionbot is a web crawler operated by Talkwalker that gathers social media and online content data for social listening, media monitoring, and consumer intelligence…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
TrendsmapResolver
TrendsmapResolver is an intelligence gatherer operated by Trendsmap. If you think this is incorrect or can provide additional detail about its purpose, please let us know…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
TTD-Content
TTD-Content is a web crawler that classifies and analyzes webpage content for programmatic advertising purposes, helping advertisers determine brand-safe placement opport…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
TurnitinBot
TurnitinBot is a web crawler operated by Turnitin, an education technology company that provides plagiarism detection and academic integrity solutions. The bot collects w…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
TweetmemeBot
TweetmemeBot is an intelligence gatherer operated by Meltwater. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Twingly
Twingly is an intelligence gatherer operated by Twingly. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Twingly Recon-Sjostrom
Twingly Recon-Sjostrom is a web crawler operated by Twingly, a company that provides social data and news data from millions of online sources including news sites, blogs…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
uipbot
uipbot is a web crawler operated by Semasio, a semantic targeting and advertising technology company that analyzes web content to understand its meaning and context for d…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
um-FC
um-FC is a web crawler operated by Ubermetrics Technologies, a company that develops solutions for media monitoring and communications intelligence. This bot collects pub…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
um-IC
um-IC is a web crawler operated by Ubermetrics Technologies, the research and development arm of the UNICEPTA Group. The bot collects web content to support advanced solu…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
um-LN
um-LN is a web crawler operated by Ubermetrics Technologies, the research and development center of the UNICEPTA Group. This bot collects web content to support AI-based …
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Verity
Verity is a web crawler operated by GumGum, a contextual advertising technology company. The bot analyzes web page content including text, images, audio, and video to ena…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
virustotal
virustotal is an intelligence gatherer operated by VirusTotal. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
weborama-fetcher
weborama-fetcher is a web crawler operated by Weborama, an AdTech company specializing in data-driven digital advertising solutions. This bot collects information from we…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
webspidermount
webspidermount is a professional job wrapping service that extracts job listings from employer websites and distributes them to job boards, recruitment platforms, and job…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
WebTrackrCrawler
WebTrackrCrawler is Affsignal's web crawler that gathers intelligence across the web to support affiliate marketing tracking and analysis.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
webzio
webzio is a web crawler operated by Webz.io that collects data from the open, deep, and dark web including news articles, blogs, forums, reviews, and social media content…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
wknd-bot
wknd-bot is operated by Wunderkind, a performance marketing platform that identifies website visitors and tracks behavioral signals. This bot is likely used for server-si…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
YaDirectFetcher
YaDirectFetcher is a web crawler operated by Yandex, the Russian search engine and technology company. It gathers data for Yandex.Direct advertising platform to analyze w…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Yahoo Ad monitoring
Yahoo Ad monitoring crawls webpages where Yahoo advertisements are served to monitor content quality, ad placement compliance, and advertiser policy adherence.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Yahoo Japan
Yahoo Japan is an intelligence gatherer operated by Yahoo. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
YaK
YaK is a web crawler operated by Linkfluence, a company that provides social media intelligence and analytics services. The bot collects web content for market research, …
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Yext Inc
Yext Inc is an intelligence gatherer operated by Yext. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
YextBot
YextBot is a web crawler operated by Yext that helps customers retrieve and organize content from their websites to maintain consistent brand information across digital c…
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
ZoominfoBot
ZoominfoBot crawls corporate websites, press releases, SEC filings and online sources using natural language processing to gather business and professional data.
Intelligence Gatherer
Analyzes web content for brand safety, competitive insights, and ad targeting
See More →
Scrapers
008
008 is a scraper operated by 80legs. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
AdsTxtCrawler
AdsTxtCrawler is a bot operated by OneTag that crawls and retrieves ads.txt files from websites. These files are part of an Interactive Advertising Bureau standard that p…
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Dataprovider.com
Dataprovider.com is a scraper operated by Dataprovider.com. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
dcrawl
dcrawl is a scraper. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
HelloworkJobPostingBot
HelloworkJobPostingBot is a web crawler that aggregates job listings by scanning company career pages and job posting websites to collect employment opportunities for the…
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
HTTrack
HTTrack is a scraper. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
HTTrack 3.0
HTTrack 3.0 is a scraper. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
IndeedJobBot
IndeedJobBot is Indeed's web scraper that crawls and indexes job postings across the internet to populate the Indeed job search platform with comprehensive employment opp…
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
KrawlerBot
KrawlerBot is a web scraping bot operated by Krawler that crawls and extracts data from websites for its customers.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
MetaInspector
MetaInspector is a scraper. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
MetaJobBot
MetaJobBot is a web scraper operated by METAJob, a job search platform based in Germany. The bot collects job listings and employment information to populate their job bo…
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
newspaper
newspaper is a scraper. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Nutch
Nutch is an open-source web crawler developed by the Apache Software Foundation. It is commonly used for large-scale web scraping.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
nyt_scraping
nyt_scraping is a web scraping bot operated by The New York Times. It is used to collect content from websites for The New York Times' purposes.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Offline Explorer
Offline Explorer is a scraper operated by MetaProducts. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
OpenindexSpider
OpenindexSpider is a scraper operated by Openindex. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Potions
Potions is a web crawler that fetches product feeds and crawls data from e-commerce websites to collect product information for commercial purposes.
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →
Scrapy
Scrapy is an open source web scraping framework written in Python that allows developers to extract data from websites. It is maintained by Zyte and used by millions of d…
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →