What is AIWebIndex?

It depends. Blocking AIWebIndex stops future fetches and keeps disallowed pages out of Lyrenth's attributed index. Lyrenth says it does not train foundation models on crawled content, but its policy allows licensing from verified domains. It does not affect traditional search rankings. For context, 7% of the top websites we track currently have robots.txt rules for AIWebIndex.

Yes. Lyrenth says AIWebIndex honors Disallow and Crawl-delay rules, limits aggregate traffic to one fetch every two seconds per domain unless Crawl-delay is longer, and treats a site as disallowed when robots.txt cannot be read and no cached copy is available. Agent Analytics can verify whether it follows those rules.

No special access. AIWebIndex can reach only content available to anonymous visitors. Make sure staging sites and other sensitive areas require authentication.

Agent Analytics tracks visits from AIWebIndex and every other known AI agent, crawler, and scraper, then authenticates each visit using AIWebIndex's published verification method. You can also check your server logs for requests whose user-agent string contains "AIWebIndex". Look for bursts that follow links across several pages and recur over time. A matching log entry alone is not proof because any bot can claim to be AIWebIndex.

AIWebIndex is adding or refreshing pages in Lyrenth's index after discovering them through sitemaps, links, or earlier fetches. Lyrenth can serve the stored pages to agents with attribution and source links.