Set Up Automatic Robots.txt for Your WordPress Website

Overview

This guide shows you how to use the WordPress plugin to set up Automatic Robots.txt. Please contact us if you need help.

Step 1: Install the Plugin

• Navigate to the Known Agents Projects page and open your project
• Copy your access token from the Settings page
• Log in to your website's WordPress dashboard
• Click Plugins in the sidebar
• Search for Known Agents (Plugin Directory)
• Click Install Now
• Click Activate

Step 2: Block Agents by Category

• Click Known Agents in the sidebar
• Paste in your access token
• Select the agent types you want to block. Allowed categories include:
  • AI Agent
  • AI Assistant
  • AI Coding Agent
  • AI Data Provider
  • AI Data Scraper
  • AI Search Crawler
  • Archiver
  • Developer Helper
  • Fetcher
  • Automated Agent
  • Intelligence Gatherer
  • Scraper
  • SEO Crawler
  • Search Engine Crawler
  • Security Scanner
  • Undocumented AI Agent
  • Uncategorized
• Check the Forcefully Block Misbehaving Bots box to enforce your rules
• Click Save Changes

Step 3: Test Your Integration

If your website is correctly connected, you should see the new rules in your website's robots.txt.