Block Bad Bots on Your Cloudflare Website (Using a Worker)

This Worker calls the Agent Identification API before forwarding each request. If the identified agent is disallowed from the requested path by your robots.txt, or if an unidentified request meets the configured automation score threshold (which you can adjust with AUTOMATION_SCORE_BLOCK_THRESHOLD), the Worker returns an HTTP 403 Forbidden response instead.

The Agent Identification API is designed for low-latency, synchronous checks. The Worker uses a three-second timeout by default, which you can adjust with FETCH_TIMEOUT_IN_MILLISECONDS.

Before continuing, set up Automatic Robots.txt for your Cloudflare website.

Step 1: Enforce Your Robots.txt and Block Unidentified Automation

• Open your domain in your Cloudflare dashboard
• In the sidebar, click Workers & Pages under Compute (Workers)
• Click the Create application button
• Select the Start with Hello World! option
• Name the worker something like known-agents-bad-bot-blocking and click Deploy
• In the top right, click the Edit code button
• Paste this code into the worker.js file:

const KNOWN_AGENTS_ACCESS_TOKEN = "YOUR_ACCESS_TOKEN"
const AUTOMATION_SCORE_BLOCK_THRESHOLD = 95
const FETCH_TIMEOUT_IN_MILLISECONDS = 3000

export default {
    async fetch(request) {
        const identification = await identifyAgent(request).catch(() => {
            return undefined
        })
        const isDisallowedByRobotsTXT = identification?.is_disallowed_by_robots_txt == true
        const isUnidentifiedAutomation =
            identification?.result == "not_identified" &&
            identification.automation_score >= AUTOMATION_SCORE_BLOCK_THRESHOLD

        if (isDisallowedByRobotsTXT || isUnidentifiedAutomation) {
            return new Response("Forbidden", {
                status: 403,
                headers: {
                    "Cache-Control": "private, no-store",
                    "Content-Type": "text/plain",
                },
            })
        }

        return fetch(request)
    },
}

async function identifyAgent(request) {
    const requestURL = new URL(request.url)
    const response = await fetch("https://api.knownagents.com/agent-identifications", {
        method: "POST",
        signal: AbortSignal.timeout(FETCH_TIMEOUT_IN_MILLISECONDS),
        headers: {
            "Authorization": `Bearer ${KNOWN_AGENTS_ACCESS_TOKEN}`,
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            request_path: requestURL.pathname + requestURL.search,
            request_headers: Object.fromEntries(request.headers),
        }),
    })

    if (!response.ok) {
        throw new Error()
    }

    return response.json()
}

• Navigate to the Known Agents Projects page and open your project
• Copy your access token from the Settings page
• Back in Cloudflare, swap in your access token where it says YOUR_ACCESS_TOKEN
• Click Deploy
• Navigate back to your Cloudflare dashboard and select your domain
• In the sidebar, click Workers Routes
• Click Add route in the HTTP Routes section
• Under Route, enter your domain followed by /* (e.g. example.com/*) and select the worker you just created
• Click Save

Tips

• Strip out any sensitive HTTP headers you don't want to send.

Step 2: Test Your Integration

• Choose an agent and path that your robots.txt disallows
• Send a request to that path using the agent's user agent
• Confirm that the response status is HTTP 403 Forbidden

Requests that are not disallowed continue to your website normally. If the Agent Identification API times out or returns an error, the Worker also allows the request through.