Face Recognition Miss; Boston Marathon Bombings

Author: Anumbor Ogor

source: CNN (Boston Marathon bombing)

The Boston marathon bombings and subsequent events that followed with the infamous Tsarnaev brothers was a missed opportunity for security agencies to identify the brothers using existing face recognition systems.

2:49 PM., April 15, 2013, Boylston community, Boston, was swarming with a beehive of people for support of those participating in a marathon race when two bombs went off close to the marathon finish line. Shock, disbelief, confusion, tears and sorrow ripped the venue after two separate explosions rocked the venue, one after the other, killing three people and injuring more than 240 people. Some of the injured people lost both limbs and others a limb, bringing it to a total of 17 people who lost limbs.

In their search, detectives compiled several video surveillance footage captured from CCTV camera feeds around the area. The videos revealed two of the brothers before the bombing, one with a round black hat and the other with a white hat. The video showed each of the brothers carrying bags on their backs walking around the venue before they were seen dropping the bags.

The whole period between occurrence of bombings till the last events surrounding the death of Tamerlan and capture of his younger brother – Dzhokhar was about 5 days interval – more specifically – 88 hours. This window period could have been more than enough for security agencies to have employed the services of a face recognition system and possibly reduced post-bombings casualty (MIT officer), but the situation was unfolding rather too quickly and the mounting pressure on security operatives to identify and capture the bombers may have been a possible reason why an opportunity of that magnitude was missed in identifying the two bombers in record time. A second theory why they may have ignored using face recognition systems during the window periods could be that the security agencies might have actually tried running but possibly they had had poor or unsatisfactory retrieval rates on their image candidate queries for the brothers.

A publication by Klontz and Jain (2013); excerpts from their article so reads:

“After reviewing photo, video, and other evidence, the FBI released images and videos of the two suspects shown. In addition to seeking identification help, the release of the images and videos was also in part to limit the damage being done to people wrongly targeted as suspects by news and social media. Shortly after the release, the two suspects were identified as brothers, Tamerlan Tsarnaev and Dzhokhar Tsarnaev, by their aunt who made a call to the FBI tip line.

“It is believed that the release of their photographs provoked the brothers into further violence, fatally shooting an MIT campus officer and carjacking a Mercedes SUV.

“These events intensified the manhunt for the brothers that ultimately ended in a violent confrontation with police officers where Tamerlan Tsarnaev was killed and Dzhokhar Tsarnaev was wounded and later captured the following day.”

In 2015, Dzhokhar Tsarnaev, 22, was sentenced to death by a federal jury. He currently waits for the date for his execution, according to CNN.

The rest of the article summarizes the work of Klontz and Jain from the published technical report. See references for report details.

 

Tsarnaev brothers

Dataset used

The dataset was a concoction of six gallery images and were added to a background set of 1.6 million mug shot photographs from the Pinellas County Sheriff’s Office (PCSO).

According to the authors, given the difficulty of automatic face detection, quality estimation, tracking, and activity recognition in uncontrolled environments, it was assumed that the law enforcement officials extracted the face images manually.

In their experiment, five probes of images were considered. The above images from the report show the Tsarnaev brothers in different poses labeled; for Tamerlan -1a and 1b, and Dzhokhar -2a, 2b, 2c, respectively, for the facial recognition experiment.

Face Matchers

Two commercial face recognition systems used in this study were NEC NeoFace 3.1 and PittPatt 5.2.2.

NeoFace was chosen based on its top performance in the National Institute of Standards and Technology (NIST) Multiple Biometrics Evaluations 2010 Test. NeoFace has strong invariance to yaw and inter-eye distance.

PittPatt 5.2.2 was selected due to its prevalent use within the law enforcement community and superior performance on non-frontal facial images. PittPatt has been acquired by Google.

Face Matching Results

Three separate results were conducted to assess the performance of the face matchers in different configurations which includes a Blind, Filtered, and Fused searches.

Blind search

In this type of search each probe is compared against all images without utilizing the demographic information (e.g. gender, ethnicity and age) related to image faces.

Blind search results - NeoFace

The PittPatt failed all probes mostly because the SDK did not allow for manual eye localization.

In the experiment, the NeoFace outperforms on all probes.

Probes for the younger brother (Dzhokhar Tsarnaev) exhibited better retrieval rates than probes for Tamerlan (elder brother) who wore sunglasses.

The figures show top three returns of each probe for NeoFace 3.1 and PittPatt 5.2.2., respectively. The sunglasses worn appear to have degraded his top matches. Demographic filtering appeared to improve the accuracy of the retrieval results.

Filtered Search

In their experiment, each probe is only compared against gallery images with similar demographic data. For Suspect 1 (white, male, 20 to 30 years old) and for suspect 2 (white, male, 15 to 25 years old), and filtering reduced the size of the gallery images from one million to 174,718 and 131,462 images, respectively.

  

Filtered search results - NeoFace

Filtered search results - PittPatt

Fused Search

Match scores using different probe images of the same suspect are summed up without weighting before ranking the gallery images.

In general, fused search improves retrieval rates for gallery images ranked similarly by each of the probes, but reported to degrade performance for gallery images ranked differently across the fused probes.

The report summarizes suggestions of more progress that is needed by research communities and tech corporations that must be made in overcoming challenges such as pose, resolution, and occlusion in order to increase the utility of unconstrained facial imagery. They conclude that with demographic filtering, multiple probes, and a human in the loop, state-of-the-art face matchers can potentially assist law enforcement in apprehending suspects in a timely fashion.

Conclusion

One year after the odd event, Facebook researchers, Taigman and his team were able to achieve accuracy of 97.35% in recognition. Their novelty DeepFace system utilized their improvised Social Face Classification dataset of large collection of Facebook photos which comprised a total of 4,030 identities and a total of 4.4 million labeled facial images in an unconstrained environment. Their DeepFace system was benchmarked against Labeled Faces in the wild database (LFW), and the YouTube Faces (YTF) dataset, for face verification in unconstrained environments.

With the advent of GPUs (Graphic Processing Units), processing power has more than doubled and GPUs have proven to be excellent choice over CPU because of their large data bandwidth for computing.

Facebook at the moment is leading the pack, but we will soon see more improvement in face recognition systems with Microsoft, Apple, IBM, Google, the FBI, others, who are not retiring or slowing down soon.

Please do well to share your thoughts below on the article, it will really be appreciated. Thank you.

About the Author; Ogor Anumbor is a mechanical engineer, a scientist-in-training, and has been studying face detection/recognition systems since 2011. He is also interested in FPGA, GPU, and image processing systems – object tracking, gait-analysis, voice recognition systems, smart-homes systems, embedded systems, and has ample field experience with microprocessors.

Contacts: eocote2002@yahoo.com, twitter.com/ogoranumbor, website: facebook.com/herculestechnology.

References

Klontz, J. C., Jain, A. K. (2013) Case Study on Unconstrained Facial Recognition Using the Boston Marathon Bombing Suspects. May 30, 2013, Technical Report-MSU-CSE-13-4.

Taigman, Y., Yang, M., Ranzato, M.A., Wolf, L. (2014) DeepFace: Closing the Gap to Humam-Level Performance in Face Verification, Facebook AI Research.

CNN, Neill, A., Cooper, A., Sanchez, R.(2015) Boston Marathon Bomber Dzhokhar Tsarnaev Sentenced to Death. May 17, 2015. http://edition.cnn.com/2015/05/15/us/boston-bombing-tsarnaev-sentence/. Date accessed: December 19, 2016. 9:30 PM.