Known Agents
Products
Agent Analytics
AI Chat Referral Tracking
Automatic Robots.txt
Agent Identification API
Setup Guides
Send Visit Events
Set Up Automatic Robots.txt
Block Bad Bots
Use the Agent Identification API
Resources
Agent Directory
The Agentic Web Index
Free Tools
AI Bot Robots.txt Checker
Pricing
Sign In
Sign Up
Menu
Products
Agent Analytics
AI Chat Referral Tracking
Automatic Robots.txt
Agent Identification API
Setup Guides
Send Visit Events
Set Up Automatic Robots.txt
Block Bad Bots
Use the Agent Identification API
Resources
Agent Directory
The Agentic Web Index
Free Tools
AI Bot Robots.txt Checker
Pricing
Sign In
Sign Up
Scrapy
Overview
Website
https://scrapy.org/
Agents & Bots
Scrapers
Scrapy
Scrapy is an open source web scraping framework written in Python that allows developers to extract data from websites. It is maintained by Zyte and used by millions of d…
Scraper
Scraper
Extracts large amounts of web data, often without explicit website permission
See More →