Known Agents
Products
Agent Analytics
AI Chat Referral Tracking
Automatic Robots.txt
Agent Identification API
Setup Guides
Send Visit Events
Set Up Automatic Robots.txt
Block Bad Bots
Use the Agent Identification API
Resources
Agent Directory
The Agentic Web Index
Free Tools
AI Bot Robots.txt Checker
Pricing
Sign In
Sign Up
Menu
Products
Agent Analytics
AI Chat Referral Tracking
Automatic Robots.txt
Agent Identification API
Setup Guides
Send Visit Events
Set Up Automatic Robots.txt
Block Bad Bots
Use the Agent Identification API
Resources
Agent Directory
The Agentic Web Index
Free Tools
AI Bot Robots.txt Checker
Pricing
Sign In
Sign Up
Scrub the Web
Overview
Website
https://scrubtheweb.com/
Agents & Bots
Search Engine Crawlers
Scrubby
Scrubby is the web crawler for Scrub the Web, an SEO-friendly business directory that has been operating since 1996. The bot crawls submitted websites to index content an…
Search Engine Crawler
Search Engine Crawler
Systematically scans and indexes web pages to include in search results
See More →