ai.robots.txt

A list of AI agents and robots to block.

What is ai.robots.txt?

A list of AI agents and robots to block.

Overview of ai.robots.txt

What it is

ai.robots.txt is a curated list of AI-related web crawlers of all types, regardless of purpose. The project provides configuration files for multiple web servers to block these crawlers, implementing the Robots Exclusion Protocol (RFC 9309). The list is community-maintained and sourced from Known Agents, and contributions are encouraged.

Key features

  • Comprehensive list of AI crawlers, including those used for training and data collection - robots.txt file implementing the Robots Exclusion Protocol - .htaccess file for Apache httpd to return an error page to listed crawlers - nginx-block-ai-bots.conf snippet for Nginx virtual host configuration - Caddyfile with a Header Regex matcher group for blocking AI bots - haproxy-block-ai-bots.txt for HAProxy configuration - lighttpd-block-ai-bots.conf for Lighttpd configuration

Use cases

  • Blocking AI crawlers from accessing website content to prevent unauthorized data collection or training - Implementing a standard robots.txt that covers a wide range of AI agents - Configuring web servers (Apache, Nginx, Caddy, HAProxy, Lighttpd) to deny requests from AI bots - Opting out of Bing AI training by adding a specific meta tag to the site's ``
  • Using the list as a reference for identifying AI crawlers in server logs or other tools

Getting started

See the repository README for detailed instructions on how to use the provided configuration files. The files can be downloaded directly from the repository and included in your web server setup.

License and pricing

The project is open source under the MIT license. It is free to use and modify.

Based on the project's public README and repository metadata.

Add this badge to your README

Featured on YunoDB Badge styles & help

Maintain this project? Show that it is featured on YunoDB.

Markdown[![Featured on YunoDB](https://yunodb.com/badge/ai-robots-txtairobotstxt.svg)](https://yunodb.com/tools/ai-robots-txtairobotstxt?utm_source=github&utm_medium=badge)
HTML<a href="https://yunodb.com/tools/ai-robots-txtairobotstxt?utm_source=github&utm_medium=badge"><img src="https://yunodb.com/badge/ai-robots-txtairobotstxt.svg" alt="Featured on YunoDB" height="20"></a>