AI is currently one of the hottest topics for those looking to invest in “the next big thing”. But according to research by the RAND Corporation, over 80% of these AI projects will fail — which is twice the failure rate for non-AI technology-related startups. The global policy think tank talked to 65 data scientists and engineers who have been working in the artificial intelligence sector over the past years, and they’ve determined several causes that lead to this massive failure rate.
According to the research, the biggest reason for the failure of AI projects is the misalignment of goals between key stakeholders. The leadership often has a view of what AI can and should achieve that is not grounded in reality; instead, it’s driven by humanity’s pre-conceived notion of what AI is, most often fuelled by Hollywood. This lack of understanding between business leaders and the people on the ground means that projects often do not have the resources and time needed to accomplish their goals.
However, the engineers working on the sharp end of AI aren’t blameless, too. The interviews revealed that data scientists sometimes get distracted by the latest developments in AI and implement them in their projects without looking at the value that it will deliver. This “shiny object syndrome” means that the scientists and engineers want to use these new technologies just because it’s the latest development. While it’s important to stay up to date on AI, teams should also consider whether that new tech would actually solve the problems they face in their research, or if it would only make it more complex and convoluted than it already is.
There are also several other reasons noted in the research, including the lack of properly prepared data sets, inadequate infrastructure, and the incompatibility of AI to the problem at hand. It also noted that these problems aren’t limited to the private sector: even the academia have issues with AI projects, where many focus simply on publishing AI research instead of looking at real-world applications for their output.
This research is proof behind the many consolidations and failures we see in the AI industry. In fact, Baidu CEO Robin Li Yanhong said China has too many large language models and that they’re wasting a significant amount of resources because these often have few, if not zero, practical real-world applications. We can also see this with the number of generative AI patents that China has filed in the past decade, outpacing the U.S. 6-to-1. But despite that, only one Chinese organization, the Chinese Academy of Sciences, made the top 20 entities that received the greatest number of citations between 2010 and 2023.
The rush to get ahead in the AI race is making many companies act a little rash in building their AI projects. While they (and their investors) are the only ones who bear the risk of any failed project, it would still be wise for them to look carefully at the failure of other AI projects and the reasons behind it. After all, if AI projects fail to deliver their promises over a long period, then the entire industry could fall and burst like a trillion-dollar bubble.
