New York City – Rachel S lives in a walkable neighbourhood in Brooklyn, New York. Most days she is able to live comfortably without a car. She works remotely often but occasionally she needs to go into the office. That’s where her situation gets a bit challenging. Her workspace is not easily accessible by public transportation.
Because she doesn’t need to drive often she applied for the car-sharing platform Zipcar to fulfill her occasional need. The application process is pretty fast allowing consumers to get on the road using its fleet of cars relatively quickly.
Unfortunately, that was not the case for Rachel. As soon as she pressed the submit button she was deemed ineligible by the artificial intelligence software the company uses. Puzzled about the outcome, Rachel got in touch with the company’s customer service team.
After all, she has no demerits that would suggest she’s an irresponsible driver. She has no points on her licence. The only flump was a traffic ticket she received when she was seventeen years old and that citation was paid off years ago.
Although the traffic citation has since been rectified, now in her thirties she is still dealing with the consequences.
She talked to Zipcar’s customer service team to no avail. Despite an otherwise clean driving record, she was rejected. She claims that the company said she had no recourse and that the decision could not be overwritten by a human.
“There was no path or process to appeal to a human being and while it is reasonable the only way to try again would be to reapply” for which there is a nonrefundable application fee, Rachel told Al Jazeera recalling her conversation with the company.
Zipcar did not respond to Al Jazeera’s request for comment.
Rachel is one of the many consumers who were declined loans, memberships and even job opportunities by AI systems without any recourse or appeal policy as companies continue to rely on AI to make key decisions that impact everyday life.
That includes D who recently lost their job.
As a condition of the interview D requested that we only use their initial out of respect for their privacy. D searched religiously for a new opportunity to no avail.
After months of looking, D finally landed a job but there was one huge problem — the timing.
It was still several weeks before D started the new job and it was several weeks after that D received the first paycheck.
To get some extra help, D applied for a personal loan on multiple platforms in an effort to circumvent predatory payday loans, just to get by in the meantime.
D was rejected for all the loans they applied for. Although D did not confirm which specific firms, the sector has multiple options including Upstart, Upgrade, SoFi, Best Egg and Happy Money, among others.
D says when they called the companies after submitting an online application, no one could help nor were there any appeals.
When D was in their early twenties they had a credit card which they failed to pay bills on. That was their only credit card. They also rent an apartment and rely on public transportation.
According to online lenders driven by AI, their lack of credit history and collateral makes them ineligible for a loan despite paying off their outstanding debt six years ago.
D did not confirm which specific companies they tried for a loan. Al Jazeera reached out to each of those companies for comment on their processes — only two responded — Upgrade and Upstart — responded by the time of publication.
“There are instances where we’re able to change the decision on the loan based on additional information, i.e. proof of other sources of income, that wasn’t provided in the original application, but when it comes to a ‘human judgment call,’ there is a lot of room for personal bias which is something regulators and industry leaders have worked hard to remove,” an Upgrade company spokesperson said in an email to Al Jazeera. “Technology has brought objectivity and fairness to the lending process, with decisions now being made based on the applicant’s true merit.”
Historical biases amplified
But it isn’t as simple as that. Existing historical biases are often amplified with modern technology. According to a 2021 investigation by the outlet The Markup, Black Americans are 80 percent more likely to be auto-rejected by loan granting agencies than their white counterparts.
“AI is just a model that is trained on historical data,” said Naeem Siddiqi, senior advisor at SAS, a global AI and data company, where he advises banks on credit risk.
That’s fueled by the United States’ long history of discriminatory practices in banking towards communities of colour.
“If you take biased data, all AI or any model will do is essentially repeat what you fed it,” Siddiqui said.
“The system is designed to make as many decisions as possible with as less bias and human judgment as possible to make it an objective decision. This is the irony of the situation… of course, there are some that fall through the cracks,” Siddiqi added.
It’s not just on the basis of race. Companies like Apple and Goldman Sachs have even been accused of systemically granting lower credit limits to women over men.
These concerns are generational as well. Siddiqi says such denials also overwhelmingly limit social mobility amongst younger generations, like younger millennials (those born between 1981 and 1996) and Gen Z (those born between 1997 and 2012), across all demographic groups.
That’s because the standard moniker of strong financial health – including credit cards, homes and cars – when assessing someone’s financial responsibility is becoming increasingly less and less relevant. Only about half of Gen Z have credit cards. That’s a decline from all generations prior.
Gen Zers are also less likely to have collateral like a car to wager when applying for a loan. According to a recent study by McKinsey, the age group is less likely to choose to get a driver’s licence than the generations prior. Only a quarter of 16-year-olds and 45 percent of 17-year-olds hold driving licences. That’s down 18 percent and 17 percent, respectively.
The Consumer Financial Protection Bureau has stepped up its safeguards for consumers. In September, the agency announced that credit lending agencies will now need to explain the reasoning behind a loan denial.
“Creditors often feed these complex algorithms with large datasets, sometimes including data that may be harvested from consumer surveillance. As a result, a consumer may be denied credit for reasons they may not consider particularly relevant to their finances,” the agency said in a release.
However, the agency does not address the lack of a human appeal process as D claims to have dealt with personally.
D said they had to postpone paying some bills which will hurt their long-term financial health and could impact their ability to get a loan with reasonable interest rates, if at all, in the future.
‘Left out from opportunities’
Siddiqi suggests that lenders should start to consider alternative data when making a decision on loans which can include rent and utility payments and even social media behavior as well as spending patterns.
On social media foreign check-ins are a key indicator.
“If you have more money, you tend to travel more or if you follow pages like Bloomberg, the Financial Times, and Reuters you are more likely to be financially responsible,” Siddiqi adds.
The auto-rejection problem is not just an issue for loan and membership applications, it’s also job opportunities. Across social media platforms like Reddit users post rejection emails they get immediately upon submitting an application.
“I fit all the requirements and hit all the keywords and within a minute of submitting my application, I got both the acknowledgement of the application and the rejection letter,” Matthew Mullen, the original poster, told Al Jazeera.
The Connecticut-based video editor says this was a first for him. Experts like Lakia Elam, head of the Human Resources consulting firm Magnificent Differences Consulting says between applicant tracking systems and other AI-driven tools, this is increasingly becoming a bigger theme and increasingly problematic.
Applicant tracking systems often overlook transferable skills that may not always align on paper with a candidate’s skill set.
“Often times applicants who have a non-linear career path, many of which come from diverse backgrounds, are left out from opportunities,” Elam told Al Jazeera.
“I keep telling organisations that we got to keep the human touch in this process,” Elam said.
But increasingly organisations are relying more on programs like ATS and ChatGPT. Elam argues that leaves out many worthwhile job applicants including herself.
“If I had to go through an AI system today, I guarantee I would be rejected,” Elam said.
She has a GED—- the high school diploma equivalency — as opposed to a four-year degree.
“They see GED on my resume and say we got to stay away from this,” Elam added.
In part, that’s why Americans do not want AI involved in the hiring process. According to an April 2023 report from Pew Research, 41 percent of Americans believe that AI should not be used to review job applications.
“It’s part of a larger conversation about losing paths to due process,” Rachel said.
