Cloudflare has announced a new feature that allows web hosting customers to block AI bots from scraping their website content without permission.
Companies responsible for AI tools, particularly those involved with training, have come under fire in recent months over their use of content published online.
Cloudflare says its new tool responds to widespread discontent among website owners and publishers about AI bots harvesting data and aims to protect content creators on the internet.
Cloudflare tool to stop AI bot scraping
In its announcement, Cloudflare wrote: “We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots.”
The tool has become available for all customers, including those on Cloudflare’s free tier.
Previously, website owners have been able to use the robots.txt file to present bots from crawling their sites, but to varying degrees of success. The new tools offers a more robust defence against content scraping by using machine learning to identify and block bots.
Cloudflare’s system relies on digital fingerprinting to differentiate bots from legitimate users.
The company said that the enhanced protection comes as a response to significant AI bot activity across its network. Bytespider, GPTBot and ClaudeBot were among the most commonly observed, but despite previous blocking attempts, AI bots accessed more than one-third (39%) of the top one million sites served by Cloudflare in June.
Thankfully, the new feature is available to all Cloudflare customers, including those on the free tier. By navigating to the Security > Bots menu and blocking “AI Scrapers and Crawlers,” users can protect their website from unwanted AI bot activity.
More from TechRadar Pro
- Microsoft AI CEO claims any content published online is "freeware" to help train AI — and people are not happy
- We’ve rounded up a list of all the best cloud hosting on offer today
- Check out the best AI writers around today
