What is micro-crawl?

Block micro-crawl if you want more control over whether your work is used for AI training. Allowing it may increase the chance that your brand appears in AI-generated answers, but either choice leaves traditional search rankings unchanged. For context, 7% of the top websites we track currently have robots.txt rules for micro-crawl.

Yes. micro-crawl is expected to follow robots.txt rules, so a disallow rule is the appropriate first step. Automatic Robots.txt can add and maintain that rule, while Agent Analytics lets you verify whether micro-crawl respects it.

Not through legitimate access. micro-crawl primarily targets public content, but some training-data scrapers also attempt to collect gated or paywalled pages. If content loads without authentication, assume it can be collected.

Agent Analytics tracks micro-crawl visits in real time. You can also check your server logs for requests whose user-agent string contains "micro-crawl". Look for high page counts, short gaps between requests, and deep traversal through linked content. Because micro-crawl does not publish a verification method, any client can claim its identity and a log match is only a clue.

Your content matched the criteria Reflection AI set for a training dataset. micro-crawl typically discovers pages through external links, sitemaps, and seed lists rather than because someone selected your site individually.