Augmented reality and artificial intelligence (AI) are usually associated with helping gamers delve deeper into the action or assisting businesses to make smart decisions. However, these technologies are also being used to create powerful tools that help people who are blind or visually impaired to become more aware of their physical surroundings.
Just how far these advances have come was brought home to Robin Spinks, strategic lead for inclusive design at the Royal National Institute of Blind People (RNIB), on a recent day out at the zoo, when his curious four-year-old son asked him what animal they were looking at.
“My first thought was it’s a black fuzzy blob,” recalls Spinks. But then he opened an app called Seeing AI, which uses AI to identify an object and then names it out loud. “I was able to point my phone’s camera at it to discover it was actually a Visayan warty pig.”
He was also able to find out more about the creature by directing his phone at the information board. “It struck me that the app was giving me the information I would have had if I was fully sighted. It was also an instant bridge between my son and I. To have that service instantly available on your phone is incredibly powerful.”
The technology, created by Microsoft in 2017, caught the eye of Haleon, a company recently spun out from GSK and which owns consumer health brands such as Sensodyne, Centrum and Voltarol. It wanted to make labels more accessible because people who don’t have full vision can struggle to differentiate between healthcare products, whether on a pharmacy shelf or in their bathroom cabinet.
The company launched a collaboration with Microsoft, and Haleon’s many products can now be recognised by the Seeing AI app when the barcode on the packaging is scanned. A bleeping noise helps the visually impaired user know when the barcode has been found. The app then reads out the name of the product, as well as its ingredients and recommended dosage. Users can skip to the information they need, so they can quickly check on dosage, for example.
For Tamara Rogers, chief marketing officer at Haleon, enabling people who are visually impaired to access the right healthcare information is crucial. “We carried out research which showed that nearly one in five visually impaired people have taken the wrong dose of a healthcare product because they were unable to read the label,” she says.
“So that’s a huge, important opportunity for us to help [make] our customers’ experience better, and safer. We’ve programmed 1,500 barcodes into Seeing AI, and we hope other brands do the same, because it’s vital we help improve accessibility to healthcare information.”
Marc Powell, accessibility innovation lead at the RNIB, who has been working with Haleon on the Seeing AI project, believes it is vital that consumer brands, public bodies and technology companies come together to empower blind and partially sighted people to read product labels and navigate their surroundings. “The Haleon work underlines how technology can help blind and partially sighted people, and with something like medication, it’s crucial people understand what they are taking and what the recommended dosage is,” he says.
The charity liaises with tech companies on the design of their apps and software to make them as inclusive as possible. Many users, for example, use the “dark mode” on a smartphone, which makes reading content easier, as do larger text options. Being able to dictate documents and have text read out by the computer or phone speaker is also popular, as is the Voice Guide feature on smart televisions, which describes the action taking place on the screen.
And the future is likely to bring further innovations in this space. “It’s a really exciting time as technology is developing so fast,” says Powell. “We’ve done work with Kellogg’s and other brands to help people locate and understand products in-store using an app called NaviLens, which is essentially the next generation of QR code that can be detected from a distance.”
The app is also being piloted in other places. “We’ve recently piloted the technology at Euston station, so that the user can detect the codes and receive information from signage, for example, in an accessible format on their smart device,” says Powell.
Wearable smart technology is another area that has huge potential. Researchers in the US are developing a chest-mounted camera and processor that sends warnings if an obstacle is ahead, through vibrating wristbands. These reveal if the impediment is to the left, right or, if both are signalling, straight ahead.
Meanwhile, another US system, called Aira, links a partially sighted or blind person to a remote operator via a headband-mounted camera. The operator can give guidance advice, based on the live video, so the user can, for example, find a cafe, have the menu explained and locate a spare table.
As for where this technological revolution is headed, Powell forecasts that wearables, such as augmented-reality glasses and ear buds, could increasingly be the way visually impaired people are helped to understand their surroundings better, through augmented-reality signs and voice prompts.
Find out how Haleon is making healthcare labelling accessible for all. Download the Microsoft Seeing AI app for free from the Apple App Store
