Robots.txt Analysis for americanthinker.com
Status
Blocked Bots
Missing AI Bots
AI Bot Coverage
Robots.txt is still highly effective. AI bots follow the rules 91.4% of the time.
4 Issues & Opportunities
-
Some AI Data Scrapers Are Not BlockedYou're blocking 13 of 50 known AI Data Scrapers, so some can still access your site.
-
Some AI Data Providers Are Not BlockedYou're blocking 1 of 29 known AI Data Providers, so some can still access your site.
-
Blocking AI Search Crawlers Is Reducing AI Search VisibilityThese agents are blocked by your robots.txt. They index and recommend pages in AI search products. Keep them blocked only if you want to limit that visibility.
-
Blocking AI Assistants Is Reducing Referral TrafficQualifiedBot is blocked by your robots.txt. They open pages to answer users and can cite or link to your site. Keep them blocked only if this is intentional.
Get a full-coverage robots.txt with Automatic Robots.txt, then track AI bot visits with Agent Analytics.
27 Blocked Bots
-
QualifiedBotAI AssistantAI AssistantFetches website content in response to a user prompt, to include in an AI-generated answer
-
DiffbotAI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service
-
AmazonbotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
Applebot-ExtendedAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
BytespiderAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
CCBotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
ClaudeBotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
CotoyogiAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
Google-ExtendedAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
GoogleOtherAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
GPTBotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
ICC-CrawlerAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
KimiBotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
meta-externalagentAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
omgiliAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
atlassian-botAI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search results
-
Google-CloudVertexBotAI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search results
-
PetalBotAI Search CrawlerAI Search CrawlerIndexes website content to possibly include as citations in AI-powered search results
-
SemrushBot-SWADeveloper HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionality
-
WARDBotDeveloper HelperDeveloper HelperAssists with testing, debugging, and ensuring website functionality
-
AwarioSmartBotIntelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targeting
-
magpie-crawlerIntelligence GathererIntelligence GathererAnalyzes web content for brand safety, competitive insights, and ad targeting
-
BaiduspiderSearch Engine CrawlerSearch Engine CrawlerSystematically scans and indexes web pages to include in search results
-
AwarioRssBotUncategorizedUncategorizedNot yet assigned a type
-
anthropic-aiUndocumented AI AgentUndocumented AI AgentCrawls websites without disclosing its purpose, collecting data for an unknown AI use case
65 Missing AI Bots
-
AgentDataBotAI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service
-
AI2BotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
Ai2Bot-DolmaAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
ApifyBotAI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service
-
ApifyWebsiteContentCrawlerAI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service
-
artsdata-crawlerAI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service
-
BeansLLM-CorpusBotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
bedrockbotAI Data ScraperAI Data ScraperDownloads website content to include in datasets used for training AI models such as LLMs
-
BixelBotAI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service
-
BravebotAI Data ProviderAI Data ProviderCrawls websites to supply structured content to AI systems as a third-party service
Frequently Asked Questions
Does robots.txt even work?
You should always have one as a first line of defense. The data shows that the vast majority of first-party AI Data Scrapers and third-party AI Data Providers actually do follow robots.txt rules.
The few bad actors willing to ignore them are usually small and have limited reach, so they probably aren't much of a threat to your business anyway.
Which AI bots should I block?
Block bots based on what they do, not simply because they use AI. Consider blocking AI Data Scrapers and AI Data Providers if you don't want your content used for model training or resale. You may also want to block general Scrapers and Undocumented AI Agents when their purpose or benefit is unclear.
Usually allow AI Search Crawlers, AI Assistants, and user-directed AI Agents if you want visibility, citations, and potential customer traffic. Keep traditional Search Engine Crawlers allowed unless you want your pages removed from their search results.
Once your rules are in place, use Agent Analytics to identify bots that ignore them, then reject their requests with Bad Bot Blocking.
How do I keep my robots.txt up to date with new AI bots?
Automatic Robots.txt adds rules for all known bots in the categories you choose and keeps them updated as new bots are discovered.
Does blocking AI bots affect my search rankings?
Blocking AI Data Scrapers and other training crawlers does not directly affect your rankings in traditional search engines. Blocking Search Engine Crawlers such as Googlebot can prevent pages from being indexed and cause them to disappear from search results. Blocking AI Search Crawlers can reduce your visibility in AI search products, even if it does not change your traditional rankings.