Preprint 2021
Alok C. Suresh and Youngjun Cho
The ability to accurately detect and model emotion is of great importance in advancing HCI applications and interfaces. Given this, the prevalence of research into the automatic emotion recognition task has steadily increased in recent years. This work is concerned with this task, particularly experimenting with attention-based CNNs on spectrograms from physiological signals. We used the spectrograms of respiration and blood volume pulse signals to condense physiological information in a two-dimensional image format. They were subsequently used to train standard CNN models as well as attention based variants for discriminating between emotional states along with various levels of valence and arousal. Also, we investigated fusion methods for the spectrograms from the two different physiological sensing channels as well as annotation strategies for classifying emotions in supervised learning. All models explored in this work were evaluated on the DEAP dataset. This work builds on our previous projects [1-7], confirming the robustness of time-frequency representation of physiological signals in automatic emotion recognition.
References
[1] Cho, Y., 2021. Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-12.
[2] Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 456-463.
[3] Cho, Y., Julier, S. J. and Bianchi-Berthouze, N., 2019. Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging. JMIR mental health, 6(4), e10140.
[4] Cho, Y. and Bianchi-Berthouze, N., 2019. Physiological and affective computing through thermal imaging: A survey. arXiv preprint arXiv:1908.10307.
[5] Cho, Y., Bianchi-Berthouze, N., Marquardt, N. and Julier, S.J., 2018. Deep thermal imaging: Proximate material type recognition in the wild through deep learning of spatial surface temperature patterns. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1-13.
[6] Cho, Y., et al.,2019. Nose heat: Exploring stress-induced nasal thermal variability through mobile thermal imaging. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 566-572
[7] Cho, Y., Julier, S.J., Marquardt, N. and Bianchi-Berthouze, N., 2017. Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomedical optics express, 8(10), pp.4480-4503.