ai.robots.txt
A list of AI agents and robots to block.
What is ai.robots.txt?
A list of AI agents and robots to block.
Overview of ai.robots.txt
What it is
ai.robots.txt is a curated list of AI-related web crawlers of all types, regardless of purpose. The project provides configuration files for multiple web servers to block these crawlers, implementing the Robots Exclusion Protocol (RFC 9309). The list is community-maintained and sourced from Known Agents, and contributions are encouraged.
Key features
- Comprehensive list of AI crawlers, including those used for training and data collection -
robots.txtfile implementing the Robots Exclusion Protocol -.htaccessfile for Apache httpd to return an error page to listed crawlers -nginx-block-ai-bots.confsnippet for Nginx virtual host configuration -Caddyfilewith a Header Regex matcher group for blocking AI bots -haproxy-block-ai-bots.txtfor HAProxy configuration -lighttpd-block-ai-bots.conffor Lighttpd configuration
Use cases
- Blocking AI crawlers from accessing website content to prevent unauthorized data collection or training - Implementing a standard
robots.txtthat covers a wide range of AI agents - Configuring web servers (Apache, Nginx, Caddy, HAProxy, Lighttpd) to deny requests from AI bots - Opting out of Bing AI training by adding a specific meta tag to the site's `` - Using the list as a reference for identifying AI crawlers in server logs or other tools
Getting started
See the repository README for detailed instructions on how to use the provided configuration files. The files can be downloaded directly from the repository and included in your web server setup.
License and pricing
The project is open source under the MIT license. It is free to use and modify.
Based on the project's public README and repository metadata.