Unless you use Google’s search engine exclusively, you probably won’t see Reddit results populate when you search for something — at least, for now.
According to a report from 404 Media, any search engine that doesn’t rely on Google’s indexing will no longer be able to crawl or scan Reddit’s site for new information. This is thanks to Reddit’s updated robots.txt file, which took effect on July 1, 2024.
Admin @traceroo on r/redditdev wrote, “Unfortunately, we’ve seen an uptick in obviously commercial entities who scrape Reddit and argue that they are not bound by our terms or policies.” Translated, Reddit doesn’t want its site crawled for data used to train or inform generative AI models that companies then charge for.
This means popular alternatives to Google search — like Microsoft Bing, DuckDuckGo, Brave, and others — will lose many search results. As someone who often tacks ‘Reddit’ onto the end of my searches in non-Google engines, that’s incredibly frustrating.
And although Reddit spokesperson Tim Rathschmidt told The Verge, “This is not at all related to our recent partnership with Google,” that’s a little hard to believe.
Is this a pay-to-access blockade from Reddit?
In February 2024, Reuters reported that Reddit signed a contract with Google worth roughly $60 million a year. This contract essentially allows Google to utilize the vast amount of human-written content available on Reddit to train its AI models.
Considering Google is the only company (so far) to strike such a deal with Reddit, it's difficult to believe Reddit's decision to update its robots.txt file isn't related to the recent partnership with Google. Even if Reddit doesn't intend to make its search results exclusive to Google, the decision is still at least somewhat related to the partnership with Google because clearly Reddit feels that it can strike similar deals with the other search engines.
In Rathschmidt's statement to The Verge, he continued, "We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them since some are unwilling or unable to make enforceable promises regarding their use of Reddit content, including their use for AI."
A Microsoft spokesperson told Search Engine Land, "We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.” But there's no mention of whether Microsoft is considering striking a deal with Reddit regarding using or strictly not using Reddit data in training generative AI models.
If no other sites sign a contract with Reddit regarding the use of its data in AI training, Google will have a huge advantage over the competition. This would be an eerie step closer to further monopolizing the search engine industry. Without popular Reddit results, the best Google Chrome alternatives just got a little less shiny.
