A Camera App that Gets to Know Your Friends

For a couple of years, Face.com has offered websites and apps a facial-recognition service that can identify people in photographs, figure out how many faces there are in the picture, which is male or female, and how old they might be. Facebook is widely believed to be one of its customers, though Face.com refuses to comment on their relationship.

But with mobile photo sharing gaining popularity, Face.com CEO Gil Hirsch says the company—which started out by building face-finding and -tagging Facebook apps—wanted to build a mobile app that, unlike existing apps that use the company's technology, would give users real-time feedback about who their cell-phone camera is pointed at. It has done so with Klik, a free smart-phone camera app with the ability to recognize faces in real time and, if it can't recognize them, learn who it is you're shooting.

Originally released in January, the latest version of Klik rolled out on Thursday for the iPhone (an Android version is coming, but Face.com won't say when). It's a bit like Instagram, but with an AI twist.

Klik connects to your Facebook account and scans tagged photos of your friends, a process that can take a few hours. Once it's ready, though, Klik can determine who you're looking at before you've pressed the shutter. It also recognizes faces in photos that are already stored on your phone.

You can take photos, dress them up with simple filters, annotate them with messages and location data, and share them on Facebook or Twitter, through e-mail, or with other Klik users.

Hirsch can think of all kinds of applications for his company's facial-recognition technology, from organizing family photos to enabling a service that could tell you more about whoever is standing in front of you.

Basically, Hirsch says, Klik picks up the presence of faces on the screen by scanning the video feed on the phone, frame by frame and pixel by pixel, searching for specific patterns it thinks make up a face. It tracks the face so it can still identify it even if it's in profile. Klik sends that visual data to the company's servers for processing, and returns with its best guess as to who's in the picture.

Going through a number of photos saved on my iPhone, Klik did a decent job at picking different friends. It did best with close-up shots, as long as there were no more than a few people in the frame, and seemed to have trouble with shots where people's eyes were closed or their faces were obscured by sunglasses or hats. It also had trouble when people were far away or not looking right into the camera.

When you focus the Klik viewfinder on a person (or several people), the name of the person Klik thinks it's seeing quickly pops up on the screen near that person's head. But if the name is incorrect—perhaps because you're not Facebook friends with them, or because you are but that person doesn't have a lot of photos of themselves on Facebook—you can hit the "learn" button to teach Klik who it is. Klik will ask you to center the person's head on the screen and then try again to determine who it is. If it still doesn't make a match, you can search for the right person in your address book or type in their name to tag a photo. The next time you take a photo of that person with Klik, it should be better able to guess their name.

Klik sometimes makes understandable mistakes. It can identify me correctly with 95 percent certainty, but it also thinks there's a chance I could be my mom, who does look like me, or an old friend from New York to whom I bear a decent resemblance.

Alessandro Acquisti, an associate professor at Carnegie Mellon University who has studied facial-recognition software, is concerned about possible privacy issues that could crop up over time if apps like Klik become more widely used and accepted. What happens, he wonders, if a Klik user's Facebook friend doesn't want to be recognized by the app? (As it turns out, Facebook's privacy settings allow you to turn off your friends' ability to share your photos with outside apps that are pulling data from Facebook.)

Acquisti does think that allowing users to correct errors is very powerful, though, and could dramatically decrease false identifications. "They're effectively enlisting the users to improve their own algorithms," he says.

A Computer Interface that Takes a Load Off Your Mind

Conversations between people include a lot more than just words. All sorts of visual and aural cues indicate each party's state of mind and make for a productive interaction.

But a furrowed brow, a gesticulating hand, and a beaming smile are all lost on computers. Now, researchers at MIT and Tufts are experimenting with a way for computers to gain a little insight into our inner world.

Their system, called Brainput, is designed to recognize when a person's workload is excessive and then automatically modify a computer interface to make it easier. The researchers used a lightweight, portable brain monitoring technology, called functional near-infrared spectroscopy (fNIRS), that determines when a person is multitasking. Analysis of the brain scan data was then fed into a system that adjusted the user's workload at those times. A computing system with Brainput could, in other words, learn to give you a break.

There are other ways that a computer could detect when a person's mental workload is becoming overwhelming. It could, for example, log errors in typing or speed of keystrokes. It could also use computer vision to detect facial expressions. "Brainput tries to get to closer to the source, by looking directly at brain activity," says Erin Treacy Solovey, a postdoctoral researcher at MIT. She presented the results last Wednesday at the Computer Human Interaction Conference in Austin, Texas.


For an experiment, Treacy Solovey and her team incorporated Brainput into virtual robots designed to adapt to the mental state of their human controller. The main goal was for each operator, capped with fNIRS headgear, to guide two different robots through a maze to find a location where a Wi-Fi signal was strong enough to send a message. But here's what made it tough: the drivers had to constantly switch between the two robots, trying to keep track of both their locations and keep them from crashing into walls.

As the research subjects drove their robots toward the strongest Wi-Fi signal, their fNIRS sensors transmitted information about their mental state to the robots. The robots, for their part, were programmed to focus on a state of mind called branching, in which a person is simultaneously working on two goals that require attention. (Previous studies have correlated certain fNIRS signals to this sort of mental state.) When the robots sensed that the driver was branching, they took on more of the navigation themselves.

The researchers found that when the robots' autonomous mode kicked in, the overall performance of the human-robot team improved. The drivers didn't seem to notice or get frustrated by the autonomous behavior of the robot when they were multitasking. The researchers also tried increasing the autonomy of the robots when Brainput did not indicate that users were mentally overloaded. When they did this, they found that overall performance decreased. In other words, increased autonomy only helped when users were struggling to cope.

"A good chunk of computer and human-computing interaction research these days is focused on giving computers better senses so they can either implicitly or explicitly augment our intellect and assist with our tasks," says Desney Tan, a researcher at Microsoft Research. "This work is a wonderful first step toward understanding our changing mental state and designing interfaces that dynamically tailor themselves so that the human-computer system can be as effective as possible."

Treacy Solovey suggests that such a system could potentially be used to help drivers, pilots, and supervisors of unmanned aerial vehicles. She says future work will investigate other cognitive states that can be reliably measured using fNIRS.