An approach for the extraction of thermal facial signatures for evaluating threat and challenge emotional states

Date of Award


Degree Name

Ph.D. in Electrical Engineering


Department of Electrical and Computer Engineering


Advisor: Vijayan K. Asari


Emotional state assessment of humans has been traditionally studied using various direct physiological measures and psychological self-reports. In this dissertation, we demonstrate that the variability in emotional and physiological states during stressful situations can be represented by analyzing indirect facial dynamics and thermo-grams. The two emotional states of challenged and threat responses in which a human under a stressful event can cope or not cope; respectively are investigated. This study uses several standoff non-invasive sensing technologies including 2D visible (VIS), Mid-Wave Infrared (MWIR) and a 3D Near Infrared (NIR) imaging cameras to examine the effect of thermal feature changes on the human face for both threat and challenge reactions. A methodology for accurately extracting facial features from thermal (MWIR) sensors and classification of these signatures against psycho-physiological ground truth are presented in this dissertation research. A threat / challenge scenario for individuals using two different human study experiments namely: false opinion study" and "false behavior study" have been created to elicit the emotional state changes. Classification studies from participants are conducted to understand the thermal facial feature changes with respect to the emotional responses. Ground truth of the threat/challenge emotional responses was conducted with direct physiological measurements and psychological self-reports which showed time-differentiated emotional response characteristics. The proposed epoch-based windowing methodology extracts statistical features from thermal facial regions to understand the small temperature changes that happen over the time-windows (e.g., epochs) of different emotions. The feature vector for representing different emotional state epochs is defined by performing Eigen analysis on the statistical features of thermal sensor data. A K-Nearest Neighbor (kNN) classifier is used to discriminate the threat and challenge emotional states. An attempt to extract variations in local intensity distributions in the visual and NIR facial region to understand the muscle variations that happen over the course of threat and challenge emotions has also been studied in this research. Region-based feature sub-selection illustrates that certain regions (e.g., forehead and nose) of the face are more useful than other regions for threat/challenge detection. Experimental evaluations performed on a set of data collected in a previously mentioned "false behavior study" show that the proposed method provides a unique and effective way to classify threat and challenge responses using thermal facial signatures. Research work is progressing to validate an optimal sensor suite outputs against established psycho-physiological variables indicative of human state responses to a stressful event. The novelty of the research is an indirect method of detecting threat/challenge emotional responses from thermal facial features that could replace direct physiological measurements for quick and non-destructive evaluation of human behavior."


Face Imaging, Infrared imaging, Emotions Data processing, Emotions Measurement, Thermal analysis, Facial expression, Electrical engineering; thermal imaging; Eigen Analysis; human state assessment

Rights Statement

Copyright © 2013, author