Set Up Automatic Robots.txt for Your Node.js Backend

Overview

This guide shows you how to use the NPM package to set up Automatic Robots.txt. Please contact us if you need help.

Step 1: Install the NPM Package

Download the package from NPM using the command line.

npm install @knownagents/sdk

Step 2: Initialize the Client

In your code, create an instance of KnownAgents with your project's access token.

import { KnownAgents } from "@knownagents/sdk"

const knownAgents = new KnownAgents("YOUR_ACCESS_TOKEN")

• Navigate to the Known Agents Projects page and open your project
• Copy your access token from the Settings page
• Back in your code, swap in your access token where it says YOUR_ACCESS_TOKEN

Step 3: Block Agents by Category

Use the generateRobotsTxt method. Select which AgentTypes you want to block, and a string specifying which URLs are disallowed (e.g. "/" to disallow all paths). Allowed agent types include:

• AI Agent
• AI Assistant
• AI Coding Agent
• AI Data Provider
• AI Data Scraper
• AI Search Crawler
• Archiver
• Developer Helper
• Fetcher
• Automated Agent
• Intelligence Gatherer
• Scraper
• SEO Crawler
• Search Engine Crawler
• Security Scanner
• Undocumented AI Agent
• Uncategorized

const robotsTXT = await knownAgents.generateRobotsTxt([
    AgentType.AIDataScraper,
    AgentType.AIDataProvider,
    AgentType.Scraper,
    AgentType.SEOCrawler
], "/")

The return value is a plain text robots.txt string.

Step 4: Serve Your Robots.txt

Generate and cache a robotsTXT periodically (e.g. once per day, using a cron job). Then, serve it from your website's /robots.txt endpoint.