In January 2020, Robert Williams was at work in an auto shop in the US city of Detroit when he received a surreal and disturbing phone call.
The caller identified himself as a police officer and told him to make his way to the nearest police station to be arrested.
Mr Williams was bewildered. He had not committed any crimes and was unaware of any reason why anyone would want to arrest him.
Dismissing the call as a prank, Mr Williams finished his workday and drove home.
When he arrived however, Mr Williams quickly realised the call was no joke.
Two Detroit police officers greeted him and, without giving any explanation, promptly arrested him for robbery.
The reason he had been arrested, Mr Williams later discovered, was because of an algorithm.
Facial recognition software had wrongly identified him as a suspect in a robbery that happened more than a year earlier in a nearby town.
This story, which was first featured by The New York Times, is one of several highlighted by New York University data journalism researcher Meredith Broussard as examples of the pitfalls of blindly relying on artificial intelligence (AI) to help solve complex problems.
"Facial recognition works better on light skin than on dark skin," Professor Broussard said.
"So, when facial recognition is used in policing, it misidentifies people of colour more often."
Speaking with the ABC News Daily podcast, Professor Broussard said this issue points to a fundamental problem — biases are being built into AI systems.
AI everywhere
In recent months AI has exploded into the mainstream, with publicly available chatbots like ChatGPT and Google Bard allowing users to do everything from build computer programs, to write catchy songs in the style of their favourite pop icons.
But according to Professor Broussard, the kind of technology used in these systems is already incorporated into many aspects of our life.
"You use AI many times a day without even realising it," she said.
"When you do a Google search you're using AI, when you record a Zoom call and have it generate a transcript automatically, you're using AI."
While generating a Zoom transcript might be very useful, Professor Broussard said some of the ways AI is used can have concerning consequences.
Police departments in the United States are already using so-called "predictive policing" systems which identify people likely to be involved in crime before an offence has even been committed.
Such systems have been shown to concentrate policing in lower socio-economic areas, perpetuating systemic racism.
Another example Professor Broussard points to is an investigation into mortgage approvals which showed algorithms were 40 to 80 per cent more likely to deny borrowers of colour compared to their white counterparts.
The reason, Professor Broussard points out, is that these approvals are based on data of existing mortgage approvals, which itself is affected by racial bias.
"In the United States, we have a very long history of financial discrimination based on race," Professor Broussard said.
"The people who have gotten mortgages in the past are not people of colour and so when you feed the computer with data about mortgages in the past, you just get the computer making decisions like that."
More than a data problem
Examining the flaws in the data raises an interesting question: If the data could somehow be altered to make it less biased, would AI systems produce fairer results?
Professor Broussard believes it would, but she said it's a goal that's often quite difficult to achieve in practice.
"Unfortunately, there is no such thing as better data in this case because there is no perfect world where we have a lack of financial discrimination in housing markets."
Even if the data itself could somehow be made bias-free, Professor Broussard said another problem remains: the biases of the people creating the AI systems.
"It's not the case that developers have set out to make biased software," she said.
"What happens is that we all have unconscious bias … so the unconscious bias of the creator ends up in the technological system."
"When you have small and homogeneous groups of people making technology as you do in Silicon Valley, for example, those technologies then get the collective blind spots of their creators.
"People are experiencing real harms from misuses of artificial intelligence and the way this shakes out is in bias from the real-world manifesting inside AI systems."
What's more, Professor Broussard said that these biases are not restricted to race, as the example of photo recognition technology reveals.
"It works better on men than on women and it does not take into account trans and non-binary folks at all," she said.
The issue, Professor Broussard said, lies in the assumption that AI technology will provide a superior or more neutral solution, a bias she terms "techno chauvinism".
"We imagine that AI is going to help us escape from human problems," she said.
"But all the problems of the human world are simply manifest inside AI systems."
Exposing the biases in the machine
In recent weeks, concerns about the potential consequences of AI's rapid development have rippled through the tech space.
It's led to a call from technology executives, including Tesla founder Elon Musk, to pause AI development for six months to ensure the risks the technology poses are manageable.
An open letter signed by hundreds of leading tech figures urges AI developers to "jointly develop and implement a set of shared safety protocols for advanced AI design and development".
"Advanced AI could represent a profound change in the history of life on Earth and should be planned for and managed with commensurate care and resources," the letter said.
But Professor Broussard said there's no reason to fear AI.
"There's no reason to think that it's coming to take over your job or that, you know, killer robots are going to come alive and take over the world," she said.
"None of that stuff is going to happen.
"At the most basic level, AI is math ... It's really complicated, beautiful math."
Nevertheless, she does believe it's important to interrogate the maths at the heart of AI to ensure bias is minimised as much as possible.
Key to this process, she said, is a growing field known as "algorithmic auditing" which involves AI experts examining both the algorithms being used as well as the data that is used to train them.
"Algorithmic auditing is really interesting because it allows us to open up the black box of an algorithm and test it to find out if it's being fair," she said.
"We need to examine questions like, 'Are we putting in dramatically more men than women? Have we put in trans, non-binary and gender non-conforming folks?'"
The answers to these kinds of questions, Professor Broussard said, may not always be easy to hear, but they are essential to ask.
"It's really difficult to confront the fact that an algorithm that you maybe worked on for months or years or spent millions of dollars on might be biased, it might be violating the law," she said.
In her recent book, More than a Glitch, Professor Broussard points to the example of the Los Angeles police department, which attempted to audit its predictive policing system only to discover the algorithms were too complicated to assess.
"There definitely are entire systems that we're going to have to throw out," she said.
"It's a tough realisation to have, but those are the tough conversations that we need to start having so that we can make sure that technology works for everybody."
