New Scientist is reporting that Facebook has a new algorithm that can determine who people are in photos, even if their face is obscured or they are looking away. You can read the full paper here, but the technology basically relies on the same type of things that ordinary humans use to identify people they can’t see full on: hair styles, posture, clothing and body shape.
Researchers used a People In Photo Albums (PIPA) dataset of just over 37,000 photos. Using Pose Invariant PErson Recognition technology, the system was able to identify people with 83% accuracy.
We propose a Pose Invariant PErson Recognition (PIPER) method, which uses part-level person recognizers to account for pose variations. We use poselets as our part models and train identity classifiers for each poselet. Poselets are classifiers that detect common pose patterns… examples are a hand next to a hip or head-and-shoulders in a back-facing view, or legs of a person walking sideways.
Outside of the general coolness that is the technology, why on earth does Facebook need super advanced face recognition technology? I would have expected this type of press release from a government anti-terrorist organization, not a social media site.1
So far as I can tell, there’s only one reason that Facebook would have to start eyeballing (figuratively) everyone’s photos: to get collect even more data about them. Yes, there are some theoretical benefits to the individual user. Facebook’s Moments app and the Google Photo Album both take advantage of this type of technology to improve photo grouping. However, I don’t recall a huge cry from people to have a computer handle their photo organization.
If Facebook can identify people in photos to this degree of accuracy, it allows them to start making connections between friends and family, even if people aren’t Facebook friends. Additionally, since the algorithm uses cues other than someone’s face, such as their body shape and hair, Facebook may also be able to start extrapolating data from that.
Could you end up seeing weight loss ads because the system thinks you’re overweight? Or ads for hair dye if it thinks you are going grey? Perhaps singles ads if you stop showing up in pictures of an ex? While this sounds far fetched, and maybe it is, Facebook spent a lot of money and time to develop the system, and they’re not doing it out of the theoretical goodness of their hearts.