Emotion detection in artistic creation: A multi-sensor fusion approach leveraging biomechanical cues and enhanced CNN models
Abstract
Artistic creation is a means of expressing human emotions. To intuitively capture the emotions conveyed by the artist in their works, we propose an improved CNN-based emotion detection method that incorporates biomechanical elements. Recognizing that emotions are accompanied by physiological and biomechanical responses such as heart rate variations, facial muscle activity, and speech tone fluctuations, we collect and integrate multi-sensor data, including heart rate, facial expression, and verbal expression. This information is processed through a multi-sensor signals fusion method based on an enhanced Convolutional Neural Networks (CNN), which allows for the extraction of rich and accurate emotional feature representations from the creator’s biomechanical signals. In particular, the facial muscle movements and subtle variations in speech tone, which are integral to understanding emotional states, are effectively captured and analyzed. Furthermore, we introduce a Conditioning Diffusion Model for Emotion Prediction, where emotional features, informed by biomechanical responses, serve as semantic conditions to boost the accuracy of emotion detection. This approach enables precise identification of the artist’s emotions by considering the intricate interplay of physiological and biomechanical signals. Experimental results demonstrate that our proposed method achieves an mAP score of 85.36%, an MSE score of 0.73%, and a runtime of 87 milliseconds, providing technical support for predicting the emotions of creators based on their biomechanical responses.
References
1. Liu Z, Zhang T, Yang K, et al. Emotion detection for misinformation: A review. Information Fusion. 2024; 107: 102300. doi: 10.1016/j.inffus.2024.102300
2. Nie W, Bao Y, Zhao Y, et al. Long Dialogue Emotion Detection Based on Commonsense Knowledge Graph Guidance. IEEE Transactions on Multimedia. 2024; 26: 514-528. doi: 10.1109/tmm.2023.3267295
3. Mamieva D, Abdusalomov AB, Kutlimuratov A, et al. Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features. Sensors. 2023; 23(12): 5475. doi: 10.3390/s23125475
4. Shin H, Lee B, Ku B, et al. Noisy label facial expression recognition via face-specific label distribution learning. Image and Vision Computing. 2024; 143: 104901. doi: 10.1016/j.imavis.2024.104901
5. Hung LP, Alias S. Beyond Sentiment Analysis: A Review of Recent Trends in Text Based Sentiment Analysis and Emotion Detection. Journal of Advanced Computational Intelligence and Intelligent Informatics. 2023; 27(1): 84-95. doi: 10.20965/jaciii.2023.p0084
6. Khan M, El Saddik A, Alotaibi FS, et al. AAD-Net: Advanced end-to-end signal processing system for human emotion detection & recognition using attention-based deep echo state network. Knowledge-Based Systems. 2023; 270: 110525. doi: 10.1016/j.knosys.2023.110525
7. Min C, Lin H, Li X, et al. Finding hate speech with auxiliary emotion detection from self-training multi-label learning perspective. Information Fusion. 2023; 96: 214-223. doi: 10.1016/j.inffus.2023.03.015
8. Nandini D, Yadav J, Rani A, et al. Design of subject independent 3D VAD emotion detection system using EEG signals and machine learning algorithms. Biomedical Signal Processing and Control. 2023; 85: 104894. doi: 10.1016/j.bspc.2023.104894
9. Krishnamoorthy P, Sathiyanarayanan M, Proença HP. A novel and secured email classification and emotion detection using hybrid deep neural network. International Journal of Cognitive Computing in Engineering. 2024; 5: 44-57. doi: 10.1016/j.ijcce.2024.01.002
10. Oğuz FE, Alkan A, Schöler T. Emotion detection from ECG signals with different learning algorithms and automated feature engineering. Signal, Image and Video Processing. 2023; 17(7): 3783-3791. doi: 10.1007/s11760-023-02606-y
11. Tzirakis P, Zhang J, Schuller BW. End-to-End Speech Emotion Recognition Using Deep Neural Networks. In: Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2018.
12. Zhao S, Ma Y, Gu Y, et al. An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos. Proceedings of the AAAI Conference on Artificial Intelligence. 2020; 34(01): 303-311. doi: 10.1609/aaai.v34i01.5364
13. Hao M, Cao WH, Liu ZT, et al. Visual-audio emotion recognition based on multi-task and ensemble learning with multiple features. Neurocomputing. 2020; 391: 42-51. doi: 10.1016/j.neucom.2020.01.048
14. Li W, Xue J, Tan R, et al. Global-Local-Feature-Fused Driver Speech Emotion Detection for Intelligent Cockpit in Automated Driving. IEEE Transactions on Intelligent Vehicles. 2023; 8(4): 2684-2697. doi: 10.1109/tiv.2023.3259988
15. Li J, Wang X, Lv G, et al. GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion Detection. IEEE Transactions on Affective Computing. 2024; 15(1): 130-143. doi: 10.1109/taffc.2023.3261279
16. Wei D, Chen D, Huang Z, et al. An improved chaotic GWO-LGBM hybrid algorithm for emotion recognition. Biomedical Signal Processing and Control. 2024; 98: 106768. doi: 10.1016/j.bspc.2024.106768
17. Wei J, Hu G, Yang X, et al. Learning facial expression and body gesture visual information for video emotion recognition. Expert Systems with Applications. 2024; 237: 121419. doi: 10.1016/j.eswa.2023.121419
18. Han X, Chen F, Ban J. FMFN: A Fuzzy Multimodal Fusion Network for Emotion Recognition in Ensemble Conducting. IEEE Transactions on Fuzzy Systems. 2025; 33(1): 168-179. doi: 10.1109/tfuzz.2024.3373125
19. Mahfoudi MA, Meyer A, Gaudin T, et al. Emotion Expression in Human Body Posture and Movement: A Survey on Intelligible Motion Factors, Quantification and Validation. IEEE Transactions on Affective Computing. 2023; 14(4): 2697-2721. doi: 10.1109/taffc.2022.3226252
20. Straker R, Exell TA, Farana R, et al. Biomechanical responses to landing strategies of female artistic gymnasts. European Journal of Sport Science. 2021; 22(11): 1678-1685. doi: 10.1080/17461391.2021.1976842
21. Coombes SA, Higgins T, Gamble KM, et al. Attentional control theory: Anxiety, emotion, and motor planning. Journal of Anxiety Disorders. 2009; 23(8): 1072-1079. doi: 10.1016/j.janxdis.2009.07.009
22. Jelodar H, Wang Y, Orji R, et al. Deep Sentiment Classification and Topic Discovery on Novel Coronavirus or COVID-19 Online Discussions: NLP Using LSTM Recurrent Neural Network Approach. IEEE Journal of Biomedical and Health Informatics. 2020; 24(10): 2733-2742. doi: 10.1109/jbhi.2020.3001216
23. Zhang J, Zhang A, Liu D, et al. Customer preferences extraction for air purifiers based on fine-grained sentiment analysis of online reviews. Knowledge-Based Systems. 2021; 228: 107259. doi: 10.1016/j.knosys.2021.107259
24. Meng J, Dong Y, Long Y, et al. An attention network based on feature sequences for cross-domain sentiment classification. Intelligent Data Analysis. 2021; 25(3): 627-640. doi: 10.3233/ida-205130
25. Lou C, Liang B, Gui L, et al. Affective Dependency Graph for Sarcasm Detection. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval; 2021.
26. Hassan SZ, Ahmad K, Hicks S, et al. Visual Sentiment Analysis from Disaster Images in Social Media. Sensors. 2022; 22(10): 3628. doi: 10.3390/s22103628
27. Yadav A, Vishwakarma DK. A Deep Multi-level Attentive Network for Multimodal Sentiment Analysis. ACM Transactions on Multimedia Computing, Communications, and Applications. 2023; 19(1): 1-19. doi: 10.1145/3517139
28. Alfreihat M, Almousa OS, Tashtoush Y, et al. Emo-SL Framework: Emoji Sentiment Lexicon Using Text-Based Features and Machine Learning for Sentiment Analysis. IEEE Access. 2024; 12: 81793-81812. doi: 10.1109/access.2024.3382836
29. Khan Z, Fu Y. Exploiting BERT for Multimodal Target Sentiment Classification through Input Space Translation. In: Proceedings of the 29th ACM International Conference on Multimedia; 2021.
30. Zhu T, Li L, Yang J, et al. Multimodal Sentiment Analysis With Image-Text Interaction Network. IEEE Transactions on Multimedia. 2023; 25: 3375-3385. doi: 10.1109/tmm.2022.3160060
31. Das R, Singh TD. Image–Text Multimodal Sentiment Analysis Framework of Assamese News Articles Using Late Fusion. ACM Transactions on Asian and Low-Resource Language Information Processing. 2023; 22(6): 1-30. doi: 10.1145/3584861
32. Liang B, Lou C, Li X, et al. Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 2022.
33. Zhang Z, Wang L, Yang J. Weakly Supervised Video Emotion Detection and Prediction via Cross-Modal Temporal Erasing Network. In: Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2023.
34. Nie L, Li B, Du Y, et al. Deep learning strategies with CReToNeXt-YOLOv5 for advanced pig face emotion detection. Scientific Reports. 2024; 14(1). doi: 10.1038/s41598-024-51755-8
35. Sharma S, S R, Akhtar MdS, et al. Emotion-Aware Multimodal Fusion for Meme Emotion Detection. IEEE Transactions on Affective Computing. 2024; 15(3): 1800-1811. doi: 10.1109/taffc.2024.3378698
36. Gupta BB, Gaurav A, Chui KT, et al. Deep Learning-Based Facial Emotion Detection in the Metaverse. In: Proceedings of the 2024 IEEE International Conference on Consumer Electronics (ICCE); 2024.
37. Kumar A. A systematic analysis of machine learning algorithms for human emotion detection using facial expression. International conference on signal processing & communication engineering systems: SPACES-2021. 2024; 2512: 020021. doi: 10.1063/5.0112473
38. Joseph A, Carvalho S, Saldanha N, et al. Emotion Detection Based on Text and Emojis. In: Proceedings of the 2024 IEEE International Conference on Information Technology, Electronics and Intelligent Communication Systems (ICITEICS); 2024.
Copyright (c) 2025 Author(s)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright on all articles published in this journal is retained by the author(s), while the author(s) grant the publisher as the original publisher to publish the article.
Articles published in this journal are licensed under a Creative Commons Attribution 4.0 International, which means they can be shared, adapted and distributed provided that the original published version is cited.
Submit a Paper
