CNN-Transformer based emotion classification from facial expressions and body gestures

Classifying the correct emotion from different data sources such as text, images, videos, and speech has been an inspiring research area for researchers from various disciplines. Automatic emotion detection from videos and images is one of the most challenging tasks that have been analyzed using supervised and unsupervised machine learning methods. Deep learning has been also employed where the model has been trained by facial and body features using pose and landmark detectors and trackers. In this paper, facial and body features extracted by the OpenPose tool have been used for detecting basic 6, 7 and 9 emotions from videos and images by a novel deep neural network framework which combines the Gaussian mixture model with CNN, LSTM and Transformer to generate the CNN-LSTM model and CNN-Transformer model with and without Gaussian centers. The experiments which were conducted using two benchmark datasets, namely FABO and CK+, showed that the proposed transformer model with 9 and 12 Gaussian centers with video generation approach was able to achieve close to 100% classification accuracy for the FABO dataset which outperforms the other DNN frameworks for emotion detection. It reported over 90% accuracy for most combinations of features for both datasets leading to a comparable framework for video emotion classification.