Cloudflare has begun blocking artificial intelligence crawlers from accessing websites on its network unless the site owner gives explicit permission. The new default setting applies to every domain that signs up for Cloudflare’s services starting this week, and affects a large swath of internet traffic. The company provides security and performance infrastructure for about 20% of the web, according to its estimates.
The move gives publishers and website owners a clearer way to control how their content is used by AI companies, many of which have built large language models by scraping the internet for articles, images, and other material often without consent or compensation.
“If you're a content creator, and you're creating value by either selling ads, selling subscriptions, or just getting the ego of knowing people are reading your stuff all three of those things are going away in an AI-driven web,” said Cloudflare Chief Executive Matthew Prince. “We’re giving publishers a way to stop that.”
