← Back

Generate Automatic Robots.txt Rules With the REST API

Request

Call this endpoint to generate a new robots.txt with the latest rules. Append the result to your robots.txt file periodically (e.g. once per day, using a cron job).

Endpoint

HTTP Method POST
URL https://api.knownagents.com/robots-txts

Headers

Header Value
Authorization A bearer token with your project's access token (e.g. Bearer YOUR_ACCESS_TOKEN). You can get your project's access token by navigating to the Known Agents Projects page, opening your project, and opening its settings page.
Content-Type Set to application/json.

Body

Field Type Required Description
agent_types string[] Yes An array of agent types you want to block or set a rule for. Allowed agent types include:
  • "AI Agent"
  • "AI Assistant"
  • "AI Coding Agent"
  • "AI Data Provider"
  • "AI Data Scraper"
  • "AI Search Crawler"
  • "Archiver"
  • "Developer Helper"
  • "Fetcher"
  • "Automated Agent"
  • "Intelligence Gatherer"
  • "Scraper"
  • "SEO Crawler"
  • "Search Engine Crawler"
  • "Security Scanner"
  • "Undocumented AI Agent"
  • "Uncategorized"
excluded_agent_tokens string[] No An array of user agent tokens whose agents should be excluded.
excluded_operator_names string[] No An array of operator names whose agents should be excluded.
disallow string No A string specifying which URLs are disallowed. Defaults to / to disallow all URLs.

Example

curl -X POST https://api.knownagents.com/robots-txts \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
        "agent_types": [
            "AI Data Scraper",
            "AI Data Provider",
            "Scraper",
            "SEO Crawler"
        ],
        "disallow": "/"
    }'

Response

The endpoint returns a string in text/plain format.

Example

User-agent: Crawl4AI
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: CCBot
Disallow: /

...