Mobile thermal imaging has created a new pathway to physiological measurements and affect recognition [1]. It enables contactless, simultaneous measurements of multiple vital signs and affective states. This can benefit healthcare, fitness sectors and specific user groups; for example, people with chronic pain who tend to reduce the number of clothes and/or devices that touch their body. Furthermore, a series of recent studies have demonstrated state-of-the-art performance with the use of small, light-weight, and low-cost sensing devices, which makes it more feasible and practical in supporting real-world applications [2-7]. To foster this promising research across communities, we release an open-source toolkit for Thermal Imaging-based Physiological and Affective computing (TIPA).
Full source code: https://github.com/deepneuroscience/TIPA
Realtime Demo: Google Colab
Language and Working Environment: Python and Jupyter Notebook (Anaconda distribution)
References
[1] Youngjun Cho and Nadia Bianchi-Berthouze. 2019. Physiological and Affective Computing through Thermal Imaging: A Survey. arXiv:1908.10307 [cs], http://arxiv.org/abs/1908.10307
[2] Cho, Y., Julier, S.J., Marquardt, N. and Bianchi-Berthouze, N., 2017. Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomedical optics express, 8(10), pp.4480-4503. https://doi.org/10.1364/BOE.8.004480
[3] Cho, Y., Julier, S.J. and Bianchi-Berthouze, N., 2019. Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging. JMIR mental health, 6(4), p.e10140. https://doi.org/10.2196/10140
[4] Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 456-463). IEEE.https://doi.org/10.1109/ACII.2017.8273639
[5] Cho, Y., Bianchi-Berthouze, N., Oliveira, M., Holloway, C. and Julier, S., 2019. Nose Heat: Exploring Stress-induced Nasal Thermal Variability through Mobile Thermal Imaging. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 566-572). IEEE. https://doi.org/10.1109/ACII.2019.8925453
[6] Cho, Y., Bianchi-Berthouze, N., Julier, S.J. and Marquardt, N., 2017. ThermSense: smartphone-based breathing sensing platform using noncontact low-cost thermal camera. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) (pp. 83-84). IEEE. https://doi.org/10.1109/ACIIW.2017.8272593
[7] Cho, Y., Bianchi-Berthouze, N., Marquardt, N. and Julier, S.J., 2018. Deep thermal imaging: Proximate material type recognition in the wild through deep learning of spatial surface temperature patterns. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-13). https://doi.org/10.1145/3173574.3173576