Cloudflare has begun blocking artificial intelligence crawlers from accessing websites on its network unless the site owner gives explicit permission. The new default setting applies to every domain that signs up for Cloudflare’s services starting this week, and affects a large swath of internet traffic. The company provides security and performance infrastructure for about 20% of the web, according to its estimates.
The move gives publishers and website owners a clearer way to control how their content is used by AI companies, many of which have built large language models by scraping th
