TalentZoo.com |  Beyond Madison Avenue |  Flack Me |  Beneath the Brand Archives  |  Categories
Facebook Begins Using Artificial Intelligence to Describe Photos to Blind Users
By: The Verge
Bookmark and Share Subscribe to the Digital Pivot RSS Feed Share
Ask a member of Facebook’s growth team what feature played the biggest role in getting the company to a billion daily users, and they’ll likely tell you it was photos. The endless stream of pictures, which users have been able to upload since 2005, a year after Facebook’s launch, makes the social network irresistible to a global audience. It’s difficult to imagine Facebook without photos. Yet for millions of blind and visually impaired people, that’s been the reality for over a decade.

Not anymore. Today Facebook will begin automatically describing the content of photos to blind and visually impaired users. Called "automatic alternative text," the feature was created by Facebook’s 5-year-old accessibility team. Led by Jeff Wieland, a former user researcher in Facebook’s product group, the team previously built closed captioning for videos and implemented an option to increase the default font size on Facebook for iOS, a feature 10 percent of Facebook users take advantage of.

Automatic alt text, which is coming to iOS today and later to Android and the web, recognizes objects in photos using machine learning. Machine learning helps to build artificial intelligences by using algorithms to make predictions. If you show a piece of software enough pictures of a dog, for example, in time it will be able to identify a dog in a photograph. Automatic alt text identifies things in Facebook photos, then uses the iPhone’s VoiceOver feature to read descriptions of the photos out loud to users. While still in its early stages, the technology can reliably identify concepts in categories including transportation ("car," "boat," "airplane"), nature ("snow," "ocean," "sunset"), sports ("basketball court"), and food ("sushi"). The technology can also describe people ("baby," "smiling," beard"), and identify a selfie.

Last week, I traveled to Facebook’s accessibility lab in Menlo Park to see the technology in action. Wieland was there, along with Matt King, a Facebook engineer who is blind. King, who was born with limited sight and became blind in college, has been advocating for more accessible computers since the 1980s. Today, he represents Facebook on a World Wide Web consortium responsible for the technical specifications that make web pages accessible.

The primary way that blind people access the internet is through a screen reader — software that describes the elements displayed on a screen (a link, a button, some text, and so on) and makes it possible to interact with them. The web has evolved over the years to be friendlier to blind people. For example, the downward-facing triangle you see on every Facebook post, which allows you to hide the post or report it as spam, gets described by the screen reader not as a triangle but as as "story options, collapsed pop-up button." That way, blind users know they can interact with it.


Bookmark and Share Subscribe to the Digital Pivot RSS Feed Share
blog comments powered by Disqus
About the Author
This article was published on The Verge. A link to the original article can be found after the post.
Digital Pivot on

Advertise on Digital Pivot
Return to Top