Is it a can of beans or tomatoes? It’s easy if you can see the label – but imagine if you were blind or partially sighted – could you tell the difference? And now consider, how would you distinguish between painkillers or other tablets without being able to see or read? Nearly one in five people have mistakenly taken a wrong dose because they haven’t been able to read the packaging, research by the consumer health company Haleon suggests.
“As humans, we like to be able to understand the world around us,” says Saqib Shaikh, an engineering manager and product leader at Microsoft who has created technology that helps decipher the visual and written world for people who are blind or have low vision.
Shaikh, who became blind at the age of seven, developed an app for Microsoft called Seeing AI to help blind people navigate the everyday. When a phone camera is pointed at an object, the app uses artificial intelligence (AI) to audibly describe it. It can narrate real-life situations, such as describing someone walking past, for example, and read out text, such as a menu.
Now Shaikh has turned his attention to the issue of health, and the challenges around packaging. He’s helped the consumer health company Haleon to tailor the Seeing AI app – first launched in 2017 – to work smoothly with its health and wellbeing products. By pointing a phone at a barcode on an item, an individual can now listen to a description and easily find information about dose, flavour or how to use a product. “No one wants to wait around for help to read the pack of painkillers or a healthcare product,” he says.
Much effort has gone on behind the scenes to upload specific information for 1,500 products into a database, says Tamara Rogers, chief marketing officer at Haleon. The company’s research reveals that more than nine in 10 (94%) people feel it is important that everyone can access the information on labels and instructions on health products, and that literacy is a significant barrier.
“Medical language is pretty challenging and not everyone can read or follow complicated information and instructions,” says Rogers.
She adds: “We want to play our part for inclusivity. The ability to manage your health in an independent way is important. We set ourselves a goal to reach 50 million people by 2025 and support them towards better everyday health.”
While the Seeing AI app already reads out text, its latest refinement makes it simpler to find the information. Packaging often doesn’t read like a book, and text can be jumbled. With audio signals, the app guides users towards the barcode on product packaging in order to align the code with the device camera, and provides key details with a simple tap. “We’ve made it easier to jump between different types of information,” says Shaikh.
And while the technology originally emerged to assist blind people, it’s also proved a lifeline for people who struggle to read or don’t have English as their first language, says Rogers. In the UK, some 2 million people are blind or have low vision, NHS figures show, and more than 8.5 million struggle to read, according to the National Literacy Trust. “The app speaks to you – you can use it in the home or when shopping,” says Rogers. “There are a whole load of folk who are carers, or visually impaired, or who can’t read smallprint who will benefit.”
Launched on 13 October to coincide with World Sight Day, the updated app is now freely available on Apple devices, and Microsoft is also taking the important step to make Seeing AI available to Android users in the future.
The collaboration between Microsoft and Haleon is a milestone on the road towards greater inclusivity, says Shaikh. Over the years, he’s experienced first-hand the freedom technology brings – from the classroom computer that first read out text, to his own software today. “I saw at school how technology could improve lives, and help level the playing field.”
When he created Seeing AI as a side-project while working at Microsoft, he had the insight to know what was needed. It can help people distinguish between denominations of banknotes, describe a high street and recognise a friend. It can also describe images, which in turn enables blind people to take a fuller part in social chit-chat online. “And that helps with social inclusion,” says Shaikh. “Now we are really using the cutting edge of AI and augmented reality to help people with disabilities.”
He continues: “Think of this technology like a little friend on your shoulder, whispering in your ear as you go about your life. That’s the power of AI. We’re taking the latest developments in technology and science to tackle problems that people meet in their daily lives. It gives you autonomy, independence.”
Haleon aims to roll the app out to more international markets and other languages. As open-source technology, other consumer goods companies could eventually upload their brands, says Rogers. “And then it will become an even better resource for those who are visually impaired or [have] literacy issues.”
Find out how Haleon is making healthcare labelling accessible for all. Download the Microsoft Seeing AI app for free from the Apple App Store
