Three-dimensional convolutional neural network based on spatial-spectral feature pictures learning for decoding motor imagery electroencephalography signal_Journal of Biomedical Engineering

Authors：

WU Xuejian ^1,2 ,  CHU Yaqi ^1,2 , ZHAO Xingang ^1,2 , ZHAO Yiwen ^1,2

1. State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, P. R. China;
2. University of Chinese Academy of Sciences (UCAS), Beijing 100049, P. R. China;

Corresponding author：

CHU Yaqi, Email: chuyaqi@sia.cn

Keywords：

Motor imagery electroencephalography; Brain-computer interface; Spatial-spectral feature picture; Feature selection; Signal decoding

DOI：

10.7507/1001-5515.202407038

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The brain-computer interface (BCI) based on motor imagery electroencephalography (EEG) shows great potential in neurorehabilitation due to its non-invasive nature and ease of use. However, motor imagery EEG signals have low signal-to-noise ratios and spatiotemporal resolutions, leading to low decoding recognition rates with traditional neural networks. To address this, this paper proposed a three-dimensional (3D) convolutional neural network (CNN) method that learns spatial-frequency feature maps, using Welch method to calculate the power spectrum of EEG frequency bands, converted time-series EEG into a brain topographical map with spatial-frequency information. A 3D network with one-dimensional and two-dimensional convolutional layers was designed to effectively learn these features. Comparative experiments demonstrated that the average decoding recognition rate reached 86.89%, outperforming traditional methods and validating the effectiveness of this approach in motor imagery EEG decoding.

Citation： WU Xuejian, CHU Yaqi, ZHAO Xingang, ZHAO Yiwen. Three-dimensional convolutional neural network based on spatial-spectral feature pictures learning for decoding motor imagery electroencephalography signal. Journal of Biomedical Engineering, 2024, 41(6): 1145-1152. doi: 10.7507/1001-5515.202407038 Copy

1.	Chen J, Xia Y, Zhou X, et al. fNIRS-EEG BCIs for motor rehabilitation: a review. Bioengineering, 2023, 10(12): 1393.
2.	Gao N, Chen P, Liang L. BCI-VR-based hand soft rehabilitation system with its applications in hand rehabilitation after stroke. Int J Precis Eng Man, 2023, 24(8): 1403-1424.
3.	Chen X, Zhao B, Wang Y, et al. Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm. J Neural Eng, 2019, 16(2): 026012.
4.	何峰, 董博文, 韩锦, 等. 基于头皮脑电的游戏型脑机接口应用研究综述. 电子与信息学报, 2022, 44(2): 415-423.
5.	Deng T, Huo Z, Zhang L, et al. A VR-based BCI interactive system for UAV swarm control. Biomed Signal Process Control, 2023, 85: 104944.
6.	迟新一, 崔红岩, 陈小刚. 结合稳态视觉诱发电位的多模态脑机接口研究进展. 中国生物医学工程学报, 2022, 41(2): 204-213.
7.	刘拓, 叶阳阳, 王坤, 等. 运动想象脑电信号分类算法的研究进展. 生物医学工程学杂志, 2021, 38(5): 995-1002.
8.	Savić A M, Lontis E R, Mrachacz-Kersting N, et al. Dynamics of movement-related cortical potentials and sensorimotor oscillations during palmar grasp movements. Eur J Neurosci, 2020, 51(9): 1962-1970.
9.	Annaby M H, Said M H, Eldeib A M, et al. EEG-based motor imagery classification using digraph Fourier transforms and extreme learning machines. Biomed Signal Process Control, 2021, 69: 102831.
10.	Malan N S, Sharma S. Motor imagery EEG spectral-spatial feature optimization using dual-tree complex wavelet and neighbourhood component analysis. IRBM, 2022, 43(3): 198-209.
11.	Zhang S, Zhu Z, Zhang B, et al. Overall optimization of CSP based on ensemble learning for motor imagery EEG decoding. Biomed Signal Process Control, 2022, 77: 103825.
12.	Echtioui A, Zouch W, Ghorbel M, et al. Classification of BCI multiclass motor imagery task based on artificial neural network. Clin EEG Neurosci, 2024, 55(4): 455-464.
13.	Amira E, Wassim Z, Mohamed G, et al. Convolutional neural network with support vector machine for motor imagery EEG signal classification. Multimed Tools Appl, 2023, 82(29): 45891-45911.
14.	Yacine F, Salah H, Amar K, et al. A novel ANN adaptive Riemannian-based kernel classification for motor imagery. Biomed Phys Eng Express, 2022, 9(1): 015010.
15.	Wang W, Qi F, David W, et al. Sparse Bayesian learning for end-to-end EEG decoding. IEEE Trans Pattern Anal Mach Intell, 2023, 45(12): 15632-15649.
16.	Mehrdad R, Salimi J S. CTRAN: CNN-Transformer-based network for natural language understanding. Eng Appl Artif Intell, 2023, 126: 107013.
17.	郭佳霖, 智敏, 殷雁君, 等. 图像处理中 CNN 与视觉 Transformer 混合模型研究综述. 计算机科学与探索, 2024: 1-18.
18.	Li H, Ding M, Zhang R, et al. Motor imagery EEG classification algorithm based on CNN-LSTM feature fusion network. Biomed Signal Process Control, 2022, 72: 103342.
19.	褚亚奇, 朱波, 赵新刚, 等. 基于时空特征学习卷积神经网络的运动想象脑电解码方法. 生物医学工程学杂志, 2021, 38(1): 1-9.
20.	Zhang R, Chen Y, Xu Z, et al. Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network. Front Neurosci, 2023, 17: 1129049.
21.	Roy A M. An efficient multi-scale CNN model with intrinsic feature integration for motor imagery EEG subject classification in brain-machine interfaces. Biomed Signal Process Control, 2022, 74: 103496.
22.	Zhang D, Chen K, Jian D, et al. Motor imagery classification via temporal attention cues of graph embedded EEG signals. IEEE J Biomed Health, 2020, 24(9): 2570-2579.
23.	Goldberger A L, Amaral L A N, Glass L, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 2000, 101(23): e215-e220.
24.	Altan G, Yayık A, Kutlu Y. Deep learning with ConvNet predicts imagery tasks through EEG. Neural Process Lett, 2021, 53(4): 2917-2932.
25.	Woodworth B E, Patel K K, Srebro N. Minibatch vs local SGD for heterogeneous distributed learning. Adv Neural Inf Process Syst, 2020, 33: 6281-6292.
26.	Antony M J, Sankaralingam B P, Mahendran R K, et al. Classification of EEG using adaptive SVM classifier with CSP and online recursive independent component analysis. Sensors, 2022, 22(19): 7596.
27.	Li M, Ruan Z. A novel decoding method for motor imagery tasks with 4D data representation and 3D convolutional neural networks. J Neural Eng, 2021, 18(4): 046029.
28.	Lawhern J V, Solon J A, Waytowich R N, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
29.	Chaudhary S, Taran S, Bajaj V, et al. Convolutional neural network based approach towards motor imagery tasks EEG signals classification. IEEE Sens J, 2019, 19(12): 4494-4500.
30.	Hermosilla D M, Codorniú R T, Baracaldo R L, et al. Shallow convolutional network excel for classifying motor imagery EEG in BCI applications. IEEE Access, 2021, 9: 98275-98286.

1. Chen J, Xia Y, Zhou X, et al. fNIRS-EEG BCIs for motor rehabilitation: a review. Bioengineering, 2023, 10(12): 1393.
2. Gao N, Chen P, Liang L. BCI-VR-based hand soft rehabilitation system with its applications in hand rehabilitation after stroke. Int J Precis Eng Man, 2023, 24(8): 1403-1424.
3. Chen X, Zhao B, Wang Y, et al. Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm. J Neural Eng, 2019, 16(2): 026012.
4. 何峰, 董博文, 韩锦, 等. 基于头皮脑电的游戏型脑机接口应用研究综述. 电子与信息学报, 2022, 44(2): 415-423.
5. Deng T, Huo Z, Zhang L, et al. A VR-based BCI interactive system for UAV swarm control. Biomed Signal Process Control, 2023, 85: 104944.
6. 迟新一, 崔红岩, 陈小刚. 结合稳态视觉诱发电位的多模态脑机接口研究进展. 中国生物医学工程学报, 2022, 41(2): 204-213.
7. 刘拓, 叶阳阳, 王坤, 等. 运动想象脑电信号分类算法的研究进展. 生物医学工程学杂志, 2021, 38(5): 995-1002.
8. Savić A M, Lontis E R, Mrachacz-Kersting N, et al. Dynamics of movement-related cortical potentials and sensorimotor oscillations during palmar grasp movements. Eur J Neurosci, 2020, 51(9): 1962-1970.
9. Annaby M H, Said M H, Eldeib A M, et al. EEG-based motor imagery classification using digraph Fourier transforms and extreme learning machines. Biomed Signal Process Control, 2021, 69: 102831.
10. Malan N S, Sharma S. Motor imagery EEG spectral-spatial feature optimization using dual-tree complex wavelet and neighbourhood component analysis. IRBM, 2022, 43(3): 198-209.
11. Zhang S, Zhu Z, Zhang B, et al. Overall optimization of CSP based on ensemble learning for motor imagery EEG decoding. Biomed Signal Process Control, 2022, 77: 103825.
12. Echtioui A, Zouch W, Ghorbel M, et al. Classification of BCI multiclass motor imagery task based on artificial neural network. Clin EEG Neurosci, 2024, 55(4): 455-464.
13. Amira E, Wassim Z, Mohamed G, et al. Convolutional neural network with support vector machine for motor imagery EEG signal classification. Multimed Tools Appl, 2023, 82(29): 45891-45911.
14. Yacine F, Salah H, Amar K, et al. A novel ANN adaptive Riemannian-based kernel classification for motor imagery. Biomed Phys Eng Express, 2022, 9(1): 015010.
15. Wang W, Qi F, David W, et al. Sparse Bayesian learning for end-to-end EEG decoding. IEEE Trans Pattern Anal Mach Intell, 2023, 45(12): 15632-15649.
16. Mehrdad R, Salimi J S. CTRAN: CNN-Transformer-based network for natural language understanding. Eng Appl Artif Intell, 2023, 126: 107013.
17. 郭佳霖, 智敏, 殷雁君, 等. 图像处理中 CNN 与视觉 Transformer 混合模型研究综述. 计算机科学与探索, 2024: 1-18.
18. Li H, Ding M, Zhang R, et al. Motor imagery EEG classification algorithm based on CNN-LSTM feature fusion network. Biomed Signal Process Control, 2022, 72: 103342.
19. 褚亚奇, 朱波, 赵新刚, 等. 基于时空特征学习卷积神经网络的运动想象脑电解码方法. 生物医学工程学杂志, 2021, 38(1): 1-9.
20. Zhang R, Chen Y, Xu Z, et al. Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network. Front Neurosci, 2023, 17: 1129049.
21. Roy A M. An efficient multi-scale CNN model with intrinsic feature integration for motor imagery EEG subject classification in brain-machine interfaces. Biomed Signal Process Control, 2022, 74: 103496.
22. Zhang D, Chen K, Jian D, et al. Motor imagery classification via temporal attention cues of graph embedded EEG signals. IEEE J Biomed Health, 2020, 24(9): 2570-2579.
23. Goldberger A L, Amaral L A N, Glass L, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 2000, 101(23): e215-e220.
24. Altan G, Yayık A, Kutlu Y. Deep learning with ConvNet predicts imagery tasks through EEG. Neural Process Lett, 2021, 53(4): 2917-2932.
25. Woodworth B E, Patel K K, Srebro N. Minibatch vs local SGD for heterogeneous distributed learning. Adv Neural Inf Process Syst, 2020, 33: 6281-6292.
26. Antony M J, Sankaralingam B P, Mahendran R K, et al. Classification of EEG using adaptive SVM classifier with CSP and online recursive independent component analysis. Sensors, 2022, 22(19): 7596.
27. Li M, Ruan Z. A novel decoding method for motor imagery tasks with 4D data representation and 3D convolutional neural networks. J Neural Eng, 2021, 18(4): 046029.
28. Lawhern J V, Solon J A, Waytowich R N, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
29. Chaudhary S, Taran S, Bajaj V, et al. Convolutional neural network based approach towards motor imagery tasks EEG signals classification. IEEE Sens J, 2019, 19(12): 4494-4500.
30. Hermosilla D M, Codorniú R T, Baracaldo R L, et al. Shallow convolutional network excel for classifying motor imagery EEG in BCI applications. IEEE Access, 2021, 9: 98275-98286.

Journal of Biomedical Engineering

Three-dimensional convolutional neural network based on spatial-spectral feature pictures learning for decoding motor imagery electroencephalography signal

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content