A face in the crowd: why facial recognition doesn't work like in the movies

If you’ve watched any sort of spy thriller or action film over the last few years – think Jason Bourne or Mission: Impossible – the chances are you’ve seen facial recognition software in action. These movie scenes often involve an artist’s sketch compared to mug shots, or sometimes even a live CCTV stream, and with the clock ticking, a match is usually found for the culprit in the nick of time.

It seems natural then to assume that what happens in the film world is similar to what happens (most of the time) in the real world. We might think that our faces are constantly being tracked and recognised as we walk past security cameras in city centres – but this is not actually the case.

Not only would such a system require millions of cameras capable of producing high-quality footage, but it would also require the integration of photo-ID databases such as mugshots from every police force, previous passport images, and driving license images for everyone in the country.

And yet even if this high level of integration was possible, a far more basic problem still exists – facial recognition systems are still not 100% accurate.

Humans on top

In relatively easy situations, where faces are front-on and reasonably clear, computer algorithms can now outperform humans. But in much harder conditions when faces are pictured from different angles or in poor lighting, humans still come out on top.

A lot of this is because of how these types of systems work in a real-world setting. Generally a computer face recognition system will provide a “candidate list” of best guesses, and from this list a human operator makes the final decision. But because these lists tend to produce several very suitable lookalikes as the target, the operator’s choice is a tough job – with error rates as high as 50%.

For similar security systems, such as the eGates used in several airports, information on their accuracy is unavailable. But there have been several cases of people being successfully “cleared” while using their partner’s passport – which could well be a cause for concern.

Face in a crowd

When it comes to recognising a face, research shows people are near perfect when shown familiar faces but fare much worse when the faces are unfamiliar.

It seems, the difficulty arises from the enormous amount of variability we see across different images of the same person. So even minor changes – like the addition of a pair of glasses – can make us significantly worse. And most people seem to be unaware of their drop in performance.

Does the middle ID image belong to the person on the left or the right? (Answer: all three are the same person).
In general, passport officers are no better than university students when matching unfamiliar people – and research shows that training in this area doesn’t seem to make a difference – you’ve either got it or you haven’t.

But there are also large individual differences in abilities over the population – from those who are clinically “face blind” to the highly accurate “super recognisers” – who are able to do just that, be super at recognising.

Faces of the future

Looking to the future, for automatic face recognition systems there is increasing evidence that using “face averages” may improve identification rates. These averages are computer-generated images, made by combining several photographs of the same person.

Computer algorithms show an advantage when using these averages to compare to a target image. A simple smartphone system is also better at recognising a user if an average image is stored in its memory.

But it may well be that long-term, we choose to leave behind our belief that facial comparison is the solution – and instead embrace other means of identification.

This could include iris and retinal scans, fingerprints, and even signature matching – all of which are considered to be more accurate than using faces.

So as things currently stand, there’s no need to be concerned about being recognised by cameras as you walk down the high street just yet. Because for the time being, you’re still just a face in a crowd.