How Publishers Can Use Known Agents

Why Publishers Choose Known Agents

Bot Traffic Is Now Business-Critical

• Almost half of your traffic now comes from bots, including AI agents, scrapers, assistants, search crawlers, and data providers.
• Bots are scraping your content and driving up server costs. Full visibility and control over their activity is strategically necessary.
• Humans will increasingly send AI agents to your website on their behalf. If your website doesn't support them, you will lose real customers.
• AI chat is becoming a major acquisition channel like search and social. If you can't measure referrals, you can't grow them.

Everything You Need to Take Back Control

• Agent Analytics shows every AI agent and bot visiting your pages in real time. This builds data-backed leverage for content licensing.
• Automatic Robots.txt protects your content with full-coverage rules that stay current. Surprisingly, 97.8% of AI bots follow them.
• Bad Bot Detection surfaces trends in activity from unidentified, spoofed, and disallowed bots and helps you block them.
• AI Chat Referral Tracking shows which stories AI platforms amplify, so you can guide content strategy and increase referrals.

What Makes Known Agents Different?

• Known Agents often costs one-tenth as much as enterprise alternatives.
• See bot traffic and AI chat referrals together. Other platforms only show one or the other.
• Track 2,000+ known bots across 15+ categories, not just the most common AI crawlers.
• Get granular breakdowns by bot, category, and page, with two-year retention and exportable reports.
• Unlike JavaScript-based analytics, Known Agents uses server-side tracking, which is more accurate when ad blockers are enabled.
• Setup is fast and simple on any platform, without risky DNS changes. Known Agents is compatible with any existing paywall or firewall.

How to Start Your Free Trial

We're happy to offer a free trial to all potential Enterprise customers.

This process is self-serve, so feel free to connect your website immediately. You can also request a demo or contact us directly.

Step 1: Create an Account

1. Sign up with your work email
2. Create a project with your website's name and domain
3. Invite team members to help manage the project

A Known Agents account manager will see that you created an account, and remove the Free plan event limit before you hit it.

Step 2: Connect Your Website

Each option below is independent and optional. Set up only the products you want to use, in any order.

1. Start Tracking Agent Visits and AI Chat Referrals
   • Follow the Send Visit Events guide to enable Agent Analytics and AI Chat Referral Tracking.
2. Get a Full-Coverage Robots.txt
   • Follow the Set Up Automatic Robots.txt guide to enable Automatic Robots.txt and keep your rules current.
3. Start Blocking Bad Bots
   • Follow the Block Bad Bots guide to block unidentified, spoofed, and disallowed bots.

Supported Platforms

Don't see your platform? Let us know, and we'll add it for you.

Frequently Asked Questions

Products

# Known Agents (updated 1 hour ago)

User-agent: GPTBot
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: CCBot
Disallow: /

...

// Response

{
  "agent_token": "Claude-User",
  "agent_type_name": "AI Assistant",
  "operator_name": "Anthropic",
  "result": "verified"
}