Research on emotion recognition methods based on multi-modal physiological signal feature fusion_Journal of Biomedical Engineering

Authors：

ZHANG Zhiwen ^1,2 ,  YU Naigong ^1,2 , BIAN Yan ^3,4 , YAN Jinhan ^1,2

1. School of Information Science and Technology, Beijing University of Technology, Beijing 100124, P. R. China;
2. Beijing Key Laboratory of Computational Intelligence and Intelligent Systems, Beijing 100124, P. R. China;
3. School of Automation and Electrical Engineering, Tianjin Polytechnic Normal University, Tianjin 300222, P. R. China;
4. Tianjin Key Laboratory of Information Sensing and Intelligent Control, Tianjin 300222, P. R. China;

Corresponding author：

YU Naigong, Email: yunaigong@bjut.edu.cn

Keywords：

Emotion; Multi-modality; Feature fusion; Electroencephalogram; Electromyogram; Electrodermal activity

DOI：

10.7507/1001-5515.202401020

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Emotion classification and recognition is a crucial area in emotional computing. Physiological signals, such as electroencephalogram (EEG), provide an accurate reflection of emotions and are difficult to disguise. However, emotion recognition still faces challenges in single-modal signal feature extraction and multi-modal signal integration. This study collected EEG, electromyogram (EMG), and electrodermal activity (EDA) signals from participants under three emotional states: happiness, sadness, and fear. A feature-weighted fusion method was applied for integrating the signals, and both support vector machine (SVM) and extreme learning machine (ELM) were used for classification. The results showed that the classification accuracy was highest when the fusion weights were set to EEG 0.7, EMG 0.15, and EDA 0.15, achieving accuracy rates of 80.19% and 82.48% for SVM and ELM, respectively. These rates represented an improvement of 5.81% and 2.95% compared to using EEG alone. This study offers methodological support for emotion classification and recognition using multi-modal physiological signals.

1.	张迪, 万柏坤, 明东. 基于生理信号的情绪识别研究进展. 生物医学工程学杂志, 2015, 32(1): 229-234.
2.	杨捷鸿, 焦学军, 曹勇, 等. 多生理信号信息融合技术的情绪识别发展. 生物医学工程研究, 2021, 40(4): 420-427.
3.	Anurag I, Amit P, Sheth. Multimodal emotion classification. CoRR, 2019(3): 120-125.
4.	王雪霏, 严欣, 周艺. 基于生理数据测量的景观情绪体验研究综述. 城市建筑, 2023, 20(9): 144-147,159.
5.	陈恬, 狄海波. 皮肤电在情绪识别研究中的应用进展. 健康研究, 2019, 39(1): 47-49.
6.	Oh S J, Kim D K. Comparative analysis of emotion classification based on facial expression and physiological signals using deep learning. Appl Sci, 2022, 12(3): 1203-1215.
7.	Liu Y J, Yu M, Zhao G, et al. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE T Affect Comput, 2017, PP(99): 1.
8.	Arya R, Singh J, Kumar A. A survey of multidisciplinary domains contributing to affective computing. Comput Sci Rev, 2021, 40(3): 100399.
9.	Maxime M, Estefania B, Kim C, et al. Multimodal emotion perception in young and elderly patients with multiple sclerosis. Mult Scler Relat Dis, 2022, 58(1): 230-238.
10.	Sousa A, d'Aquin M, Zarrouk M, et al. Person-independent multimodal emotion detection for children with high-functioning autism. Ceur Workshop Proceedings, 2020, 2760(13): 1598-1610.
11.	Morency L P, Mihalcea R, Doshi P. Towards multi-modal sentiment analysis: harvesting opinions from the web// Proceedings of the 13th International Con-ference on Multimodal Interfaces. Alicante: IEEE, 2011: 169-176.
12.	Zadeh A, Zellers R, Pincus E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intel Sys, 2016, 31(6): 82-88.
13.	Pérez-Rosas V, Mihalcea R, Morency L P. Utterance-level multimodal sentiment analysis// Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Sofia: IEEE, 2013: 973-982.
14.	牛晓伟, 刘光远. 用于生理信号情感识别的自适应遗传算法. 计算机工程与设计, 2008(14): 3726-3728,3731.
15.	Hassan M M, Alam M G R, Uddin M Z, et al. Human emotion recognition using deep belief network architecture. Inform Fusion, 2019, 51: 10-18.
16.	李芳, 朱昭红, 白学军. 高兴和悲伤电影片段诱发情绪的有效性和时间进程. 心理与行为研究, 2009, 7(1): 32-38.
17.	Mert A, Akan A. Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl, 2018, 21(1): 1-9.
18.	Chen Z. Research on cognitive differences on cerebral left and right cognition in the processing of emotional vocabulary. AP, 2016, 6(3): 290-298.
19.	Liu J, Su Y, Liu Y. Multi-modal emotion recognition with temporal-band attention based on LSTM-RNN. Cham: Springer, 2017.
20.	凌文芬, 陈思含, 彭勇, 等. 基于3D分层卷积融合的多模态生理信号情绪识别. 智能科学与技术学报, 2021, 3(1): 76-84.
21.	Guo K, Candra H, Yu H, et al. EEG-based emotion classification using innovative features and combined SVM and HMM classifier// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju Island: IEEE, 2017: 489-492.
22.	汪佳衡, 王跃明, 姚林. 基于滤波器组长短时记忆网络的脑电信号情绪识别. 生物医学工程学杂志, 2021, 38(3): 447-454.
23.	陈东伟, 陈俊杰. 情感识别中脑电信号Lempel-Ziv复杂度的研究. 太原理工大学学报, 2014, 45(6): 758-763.
24.	陈沙利. 基于外周生理信号的情绪识别研究. 杭州: 浙江大学, 2021.
25.	邓丽娜, 王晓亮. 基于生理信号的情绪识别研究综述. 物联网技术, 2021, 11(7): 33-41.
26.	赵运基, 范存良, 张新良. 融合多特征和通道感知的目标跟踪算法. 计算机科学与探索, 2022, 16(6): 1417-1428.
27.	Alanazi S A, Alruwaili M, Ahmad F, et al. Estimation of organizational competitiveness by a hybrid of one-dimensional convolutional neural networks and self-organizing maps using physiological signals for emotional analysis of employees. Sensors (Basel), 2021, 21(11): 3760.
28.	杜思清. 基于皮肤电信号的心理应激建模的研究. 南京: 东南大学, 2019.
29.	李佳泽, 梅红岩, 贾丽云, 等. 动态时间序列建模的多模态情感识别方法. (2023-12-01) [2024-10-29]. 计算机工程与应用. https://link.cnki.net/urlid/11.2127.TP.20231130.1701.028.
30.	Domínguez-Jiménez A, Campo-Landines K C, Martínez-Santos J C, et al. A machine learning model for emotion recognition from physiological signals. Biomed Signal Proces, 2020, 55(C): 101646.
31.	Shu L, Xie J, Yang M, et al. A review of emotion recognition using physiological signals. Sensors, 2018, 18(7): 2074.

1. 张迪, 万柏坤, 明东. 基于生理信号的情绪识别研究进展. 生物医学工程学杂志, 2015, 32(1): 229-234.
2. 杨捷鸿, 焦学军, 曹勇, 等. 多生理信号信息融合技术的情绪识别发展. 生物医学工程研究, 2021, 40(4): 420-427.
3. Anurag I, Amit P, Sheth. Multimodal emotion classification. CoRR, 2019(3): 120-125.
4. 王雪霏, 严欣, 周艺. 基于生理数据测量的景观情绪体验研究综述. 城市建筑, 2023, 20(9): 144-147,159.
5. 陈恬, 狄海波. 皮肤电在情绪识别研究中的应用进展. 健康研究, 2019, 39(1): 47-49.
6. Oh S J, Kim D K. Comparative analysis of emotion classification based on facial expression and physiological signals using deep learning. Appl Sci, 2022, 12(3): 1203-1215.
7. Liu Y J, Yu M, Zhao G, et al. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE T Affect Comput, 2017, PP(99): 1.
8. Arya R, Singh J, Kumar A. A survey of multidisciplinary domains contributing to affective computing. Comput Sci Rev, 2021, 40(3): 100399.
9. Maxime M, Estefania B, Kim C, et al. Multimodal emotion perception in young and elderly patients with multiple sclerosis. Mult Scler Relat Dis, 2022, 58(1): 230-238.
10. Sousa A, d'Aquin M, Zarrouk M, et al. Person-independent multimodal emotion detection for children with high-functioning autism. Ceur Workshop Proceedings, 2020, 2760(13): 1598-1610.
11. Morency L P, Mihalcea R, Doshi P. Towards multi-modal sentiment analysis: harvesting opinions from the web// Proceedings of the 13th International Con-ference on Multimodal Interfaces. Alicante: IEEE, 2011: 169-176.
12. Zadeh A, Zellers R, Pincus E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intel Sys, 2016, 31(6): 82-88.
13. Pérez-Rosas V, Mihalcea R, Morency L P. Utterance-level multimodal sentiment analysis// Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Sofia: IEEE, 2013: 973-982.
14. 牛晓伟, 刘光远. 用于生理信号情感识别的自适应遗传算法. 计算机工程与设计, 2008(14): 3726-3728,3731.
15. Hassan M M, Alam M G R, Uddin M Z, et al. Human emotion recognition using deep belief network architecture. Inform Fusion, 2019, 51: 10-18.
16. 李芳, 朱昭红, 白学军. 高兴和悲伤电影片段诱发情绪的有效性和时间进程. 心理与行为研究, 2009, 7(1): 32-38.
17. Mert A, Akan A. Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl, 2018, 21(1): 1-9.
18. Chen Z. Research on cognitive differences on cerebral left and right cognition in the processing of emotional vocabulary. AP, 2016, 6(3): 290-298.
19. Liu J, Su Y, Liu Y. Multi-modal emotion recognition with temporal-band attention based on LSTM-RNN. Cham: Springer, 2017.
20. 凌文芬, 陈思含, 彭勇, 等. 基于3D分层卷积融合的多模态生理信号情绪识别. 智能科学与技术学报, 2021, 3(1): 76-84.
21. Guo K, Candra H, Yu H, et al. EEG-based emotion classification using innovative features and combined SVM and HMM classifier// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju Island: IEEE, 2017: 489-492.
22. 汪佳衡, 王跃明, 姚林. 基于滤波器组长短时记忆网络的脑电信号情绪识别. 生物医学工程学杂志, 2021, 38(3): 447-454.
23. 陈东伟, 陈俊杰. 情感识别中脑电信号Lempel-Ziv复杂度的研究. 太原理工大学学报, 2014, 45(6): 758-763.
24. 陈沙利. 基于外周生理信号的情绪识别研究. 杭州: 浙江大学, 2021.
25. 邓丽娜, 王晓亮. 基于生理信号的情绪识别研究综述. 物联网技术, 2021, 11(7): 33-41.
26. 赵运基, 范存良, 张新良. 融合多特征和通道感知的目标跟踪算法. 计算机科学与探索, 2022, 16(6): 1417-1428.
27. Alanazi S A, Alruwaili M, Ahmad F, et al. Estimation of organizational competitiveness by a hybrid of one-dimensional convolutional neural networks and self-organizing maps using physiological signals for emotional analysis of employees. Sensors (Basel), 2021, 21(11): 3760.
28. 杜思清. 基于皮肤电信号的心理应激建模的研究. 南京: 东南大学, 2019.
29. 李佳泽, 梅红岩, 贾丽云, 等. 动态时间序列建模的多模态情感识别方法. (2023-12-01) [2024-10-29]. 计算机工程与应用. https://link.cnki.net/urlid/11.2127.TP.20231130.1701.028.
30. Domínguez-Jiménez A, Campo-Landines K C, Martínez-Santos J C, et al. A machine learning model for emotion recognition from physiological signals. Biomed Signal Proces, 2020, 55(C): 101646.
31. Shu L, Xie J, Yang M, et al. A review of emotion recognition using physiological signals. Sensors, 2018, 18(7): 2074.

Journal of Biomedical Engineering

Latest ArticlesResearch on emotion recognition methods based on multi-modal physiological signal feature fusion

Abstract Full text Figures/Tables Video References Cited by

Format

Content