Agents
All Agent Types
Every known artificial agent (bot) on the internet. You can track their activity on your website with Agent Analytics, or control their behavior with Automatic Robots.txt.
Fetchers
MissinglettrBot
MissinglettrBot is a web crawler that automatically detects and fetches blog post content from customer websites to create automated social media drip campaigns and marke…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
MonitoRSS
MonitoRSS is a Discord bot that polls RSS feeds at regular intervals to deliver highly-customized news feeds directly to Discord channels.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
MuckRackFeedParser
MuckRackFeedParser is a feed parsing bot operated by Muck Rack, a platform that connects journalists with public relations professionals. This bot fetches and parses RSS …
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
NewsBlur
NewsBlur is a web crawler that fetches RSS feeds and web content for its personal news reader service, collecting articles from subscribed websites and blogs to deliver r…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
NewsBlur Feed Fetcher
NewsBlur Feed Fetcher is the bot used by NewsBlur, an RSS feed reader service, to retrieve and update content from RSS and Atom feeds that users have subscribed to. The b…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
NewsBlur Page Fetcher
NewsBlur Page Fetcher is a bot operated by NewsBlur, an RSS feed reader and news aggregation service. It fetches content from websites on behalf of subscribers, with the …
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Newslitbot
Newslitbot is a web crawler operated by Newslit, a media monitoring service that tracks over 100,000 news sources to provide personalized news briefs and content curation…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
NextCloud
NextCloud is a fetcher operated by Nextcloud. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Nooshub
Nooshub is a modern RSS reader bot that fetches RSS and Atom feeds and uses learning algorithms to group similar articles, remove clutter, and identify trending topics. I…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
OdklBot
OdklBot is a crawler that fetches link preview data for the Odnoklassniki social network, extracting Open Graph metadata (titles, descriptions, images) from external link…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
OpenGraph.io
OpenGraph.io is a web crawler that extracts Open Graph metadata, titles, descriptions, and images from web pages to generate rich link previews for social sharing platfor…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
OpenRSS
OpenRSS is a nonprofit RSS feed service bot that crawls websites to generate and provide free RSS feeds, allowing users to follow website updates in a reader app.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Orlo-LinkPreview
Orlo-LinkPreview bot fetches metadata and preview information from URLs that are scheduled for publication in social media posts through the Orlo social media management …
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Overcast
Overcast bot fetches RSS feeds and audio metadata from podcast hosting servers to keep the podcast directory and episodes updated for the award-winning iOS podcast player…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Ozon Web Grabber
Ozon Web Grabber is a fetcher operated by Ozon. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
PagePeeker
PagePeeker is a website thumbnail and screenshot generation service that visits web pages to capture visual previews of their content. The bot creates preview images that…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Pinterestbot
Pinterestbot is Pinterest's web crawler that fetches content from websites to create previews when users pin links to the platform. The bot collects metadata, images, and…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Piqo Podsync
Piqo Podsync is a fetcher bot operated by Piqo that synchronizes or retrieves podcast-related content. Based on its name and designation as a fetcher, it likely accesses …
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Pocket Casts Feed Parser
Pocket Casts Feed Parser fetches and parses podcast RSS feeds to deliver content for playback in the Pocket Casts mobile and web applications.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
PocketParser
PocketParser is a web crawler operated by Mozilla's Pocket service that fetches and parses web content when users save articles and videos for later reading. The bot extr…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
PodchaserParser
PodchaserParser is a web crawler operated by Podchaser, a podcast database and discovery platform. This bot fetches podcast-related information from websites to index and…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Podimo
Podimo is a podcast and audiobook streaming platform bot that fetches content from websites, likely to discover, index, or aggregate podcast episodes and audio content fo…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
PressEngine Bot
PressEngine Bot is a fetcher operated by PressEngine. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
PressEngineBot
PressEngineBot is an automated verification crawler that confirms video game press coverage URLs exist and are viewable when submitted by journalists who receive review k…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Protopage
Protopage is an RSS reader and web portal service that crawls and indexes RSS news feeds from various sources to create personalized dashboard start pages and aggregated …
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
qcbot
qcbot is a web crawler used by QUIC.cloud to fetch data from websites for various online services including image optimization, CSS optimization, and other website perfor…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Quora-Bot
Quora-Bot is a web crawler operated by Quora, a question-and-answer platform where users can ask questions and get answers from the community. This bot fetches web conten…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
rakutenusabot-image
rakutenusabot-image is a specialized web crawler used by Rakuten to crawl and fetch product images from merchant websites. This enables Rakuten to display cashback deals …
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
redditbot
redditbot is a web crawler operated by Reddit, a social news aggregation and discussion platform. This bot fetches web content to create thumbnails and rich previews when…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Reelevant
Reelevant is a web crawler that fetches product images at runtime to dynamically update content inside email campaigns for personalized marketing experiences.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
reward-gateway
reward-gateway is a web crawler operated by Reward Gateway, an employee experience platform company. The bot likely fetches content related to employee benefits, workplac…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
RSS API
RSS API is a fetcher operated by Tibush. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
RSS.Social
RSS.Social is a fetcher bot that retrieves RSS feeds, likely as part of a service that automates social media posting or manages RSS content distribution.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
rss2tg bot
rss2tg bot is an automated service that monitors RSS feeds and sends content updates to Telegram channels or users, enabling real-time notifications for feed content.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
RSSAPI
RSSAPI is a service that periodically monitors RSS, ATOM, and JSON feeds for customers, parsing and normalizing content, then delivering updates via webhooks in near real…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
scoopit-crawler
scoopit-crawler is a web crawler operated by Scoop.it, a content curation platform that helps users discover, curate, and share content on various topics. The bot fetches…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
scour.ing
scour.ing is Scour's AI-powered web crawler that fetches content from RSS feeds and websites to create personalized content recommendations based on user interests.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
ScourRSSBot
ScourRSSBot is a fetcher bot operated by Scour that retrieves RSS feed content from websites. The bot is likely used to collect and aggregate syndicated content for the S…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
serendeputybot
serendeputybot is a fetcher operated by Serendeputy. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
SFDC-Callout
SFDC-Callout is the user agent used by Salesforce when making HTTP callouts and API requests from Apex code, workflow rules, and other Salesforce automated processes to e…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
ShortPixel
ShortPixel is a bot from an image optimization service that compresses and converts images to formats like WebP and AVIF. It visits websites to fetch and optimize images …
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Shortwave Image Fetcher
Shortwave Image Fetcher is a privacy-focused service that proxies images from HTML emails through Shortwave's servers to protect users' IP addresses and maintain email pr…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
SimplePie
SimplePie is a fetcher operated by SimplePie. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Skroutz ImageBot
Skroutz ImageBot crawls and fetches product images from e-commerce websites to display in Skroutz's price comparison platform, enabling visual product browsing and compar…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
SkroutzBot
SkroutzBot is a specialized web crawler used by Skroutz to download XML data feeds, product images, and sample product pages from merchant websites for quality control an…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Skype
Skype is a fetcher operated by Skype. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
SkypeUriPreview
SkypeUriPreview is Skype's link preview bot that fetches page content when users share URLs in Skype messages, extracting metadata to display rich previews with titles, d…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Slack Image Proxy
Slack Image Proxy is a fetcher operated by Slack. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Slack-ImgProxy
Slack-ImgProxy is a web crawler used by Slack to fetch, process, and cache images that are shared in Slack channels, ensuring they can be displayed properly within the me…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →
Slackbot
Slackbot is operated by Slack and is used to fetch link previews and unfurl content from websites when users share URLs in Slack conversations. When a link is posted, thi…
Fetcher
Retrieves web page metadata to power app features like link previews or feeds
See More →