Facebook improves how blind can see images using AI

When Matt King first got on Facebook eight years ago, the blind engineer had to weigh whether it was worth spending an entire Saturday morning checking whether a friend of his was actually on his friend list. Such were the tools at the time for the visually impaired — almost nonexistent.

Today, thanks to text-to-audio software, it just takes a few seconds for him to accomplish the same task. And because of a new face recognition service the social network is rolling out Tuesday, he can now learn which friends are in photos, even those who haven’t been tagged by another user.

The facial recognition technology, which uses artificially intelligent algorithms, doesn’t appear to have changed much since Facebook began using it in 2010 to suggest the identities of people in photos. But after incorporating feedback from billions of user interactions, Facebook felt confident enough to push its use into new territory.

“What we’re doing with AI is making it possible for anybody to enjoy the experience,” says 52-year-old King, who lost his sight in college due to a degenerative eye disease and now works at Facebook as an accessibility specialist. In addition to the improved facial recognition, Facebook has in recent years also automated descriptions of what’s happening in a photo. (Those remain relatively primitive, as they’re limited to only about 100 or so concepts and roughly a dozen action verbs.)

For the sighted, the new facial recognition settings will also help crack down on imposters. Starting Tuesday, Facebook will notify you if someone has uploaded your face as their profile picture. And just in time for alcohol-laden holiday parties, you can also be notified if someone in your friend network has posted a compromising picture of you without explicitly tagging you.

That check is not perfect. If you don’t want a drunken karaoke picture of yourself online, for example, you can’t veto the resulting photos posted by your friends, although you can ask them to take it down. Serious violations or disputes can be flagged to Facebook staff, which they will review according to their community standards.

Facebook lets the person posting a photo have the final say this way in order to deal with situations like public speaking events, where the speaker shouldn’t have control over what’s shared.

For the visually impaired, the company is also working on identifying text in manipulated photos like memes, though its technology isn’t quite ready for primetime. Even small accuracy problems can ruin the punchline, according to Facebook accessibility director Jeff Wieland.

For Janni Lehrer-Stein, a volunteer who advises Facebook, the new facial recognition capability will help her enjoy more of her daughter’s recent graduation photos. Lehrer-Stein had lost her sight by the time she was 46 and has long used the audio-to-text software on Facebook. Knowing who is in each photo makes a big difference, says the sixty-something mother of three.

The technology gives her enough information to “engage” in special family moments captured in photos, she says. “Even though I can’t see them.”