基于BERT模型和动态集成选择的多分类文本情感识别研究

doi:10.16381/j.cnki.issn1003-207x.2021.1159

Abstract

Abstract:

To handle semantic deficiency of text feature vector extracted by classic methods and the issue of multi-classsentimentclassification in the text emotion recognition task， a novel multi-class sentiment classification strategy based onBidirectional Encoder Representations from Transformers （BERT） and dynamic ensemble selection （DES） is proposed. First， BERT is used to vectorize the text.Then， the OVO strategy is used to divide the multi-class sentiment classification problem into multiple binary classification sub-problems.Next， the dynamic ensemble selection strategy is developed to construct binary classifier for dealing with each sub-problem.Finally， the final prediction result is obtained based on the aggregation strategy. A public movie review data set is employed to carry out the experimental analysis. The experimental results indicate that（1） the BERT model is helpful in improving the multi-class sentiment classification performancewith respect to these traditional methods， namely TFIDF and Wor2Vec，（2） it is effective to use the DES strategy for dealing with each sub-problem in multi-class sentiment classification， and （3）the performance of the proposed method is also significantlybetter than that of the existing well-known methods for multi-class sentiment analysis.

Key words: text sentiment analysis, BERT, multi-class, dynamic ensemble selection, decomposition strategy

CLC Number:

TP391.1

Zhongliang Zhang,Qinjun Fei,Yuyu Chen,Xinggang Luo. Researchon Multi-class Sentiment Classification Based on BERT and Dynamic Ensemble Selection[J]. Chinese Journal of Management Science, 2024, 32(6): 140-150.

Figures/Tables 7

References 64

1	王婷，杨文忠.文本情感分析方法研究综述［J］.计算机工程与应用，2021，57（12）：11-24.
	Wang T， Yang W Z.Review of text sentiment analysis methods［J］.Computer Engineering and Applications， 2021，57（12）：11-24.
2	许伟，刘令宇，王明明.基于跨媒体分析的突发事件检测及趋势研判研究［J］.系统工程理论与实践，2015，35（10）：2550-2556.
	Xu Ｗ， Liu L Y， Wang M M.Emergency event detection and analysis for emergence management based on cross-media analytics［J］.Systems Engineering-Theory & Practice， 2015，35（10）：2550-2556.
3	张翼鹏，马敬东.突发公共卫生事件误导信息受众情感分析及传播特征研究［J］.数据分析与知识发现，2020，4（12）：45-54.
	Zhang Y P， Ma J D. Analyzing sentiments and dissemination of misinformation on public health emergency［J］. Data Analysis and Knowledge Discovery， 2020， 4（12）： 45-54.
4	王安宁，张强，彭张林，等.融合特征情感和产品参数的客户感知偏好模型［J］.中国管理科学，2020，28（9）：199-208.
	Wang A N， Zhang Q， Peng Z L， et al.Customer preference model considering feature sentiment and product parameters［J］.Chinese Journal of Management Science， 2020，28（9）：199-208.
5	沈超，王安宁，方钊，等.基于在线评论数据的产品需求趋势挖掘［J］.中国管理科学，2021，29（5）：211-220.
	Shen C， Wang A N， Fang Z， et al. Trend mining of product requirements from online reviews［J］. Chinese Journal of Management Science， 2021，29（5）：211-220.
6	尤天慧，张瑾，樊治平.基于情感分析和证据理论的多属性在线评论决策方法［J］.系统管理学报，2019，28（3）：536-544.
	You T H， Zhang J， Fan Z P. Multi-attribute online review decision making method based on sentiment analysis and evidence theory［J］. Journal of Systems & Management， 2019，28（3）：536-544.
7	Coutinho D P， Figueiredo M A T. Text classification using compression： Based dissimilarity measures［J］. International Journal of Pattern Recognition and Artificial Intelligence， 2015， 29（5）： 1553004.
8	钟佳娃，刘巍，王思丽，等.文本情感分析方法及应用综述［J］.数据分析与知识发现，2021，5（6）：1-13.
	Zhong J W， Liu W， Wang S L，et al. Review of methods and applications of text sentiment analysis［J］.Data Analysis and Knowledge Discovery， 2021， 5（6）： 1-13.
9	Rice D R， Zorn C. Corpus-based dictionaries for sentiment analysis of specialized vocabularies［J］. Political Science Research and Methods， 2021， 9（1）： 20-35.
10	Manek A S， Shenoy P D， Mohan M C， et al. Aspect term extraction for sentiment analysis in large movie reviews using gini index feature selection method and SVM classifier［J］. World Wide Web， 2017， 20（2）： 135-154.
11	Kolli S， Murthy N， Prasad R. Sentiment classification on online retailer reviews［M］//Kumar A， Mozar S. ICCCE 2020. Singapore： Springer， 2021： 1557-1563.
12	Syamala M， Nalini N J. A filter based improved decision tree sentiment classification model for real-time amazon product review data［J］. International Journal of Intelligent Engineering and Systems， 2020，13（1）： 191-202.
13	Yadav A， Vishwakarma D K. Sentiment analysis using deep learning architectures： A review［J］. Artificial Intelligence Review， 2020， 53（6）： 4335-4385.
14	Serrano-Guerrero J， Olivas J A， Romero F P， et al. Sentiment analysis： A review and comparative analysis of web services［J］. Information Sciences， 2015， 311： 18-38.
15	Almeida A M G， Cerri R， Paraiso E C， et al. Applying multi-label techniques in emotion identification of short texts［J］. Neurocomputing， 2018， 320： 35-46.
16	Bickerstaffe A， Zukerman I. A hierarchical classifier applied to multi-way sentiment detection［C］//Proceedings of the 23rd International Conference on Computational Linguistics， Beijing， China， 23-27 August， Association for Computational Linguistics， 2010： 62-70.
17	Liu Y， Bi J W， Fan Z P. A method for multi-class sentiment classification based on an improved one-vs-one （OVO） strategy and the support vector machine （SVM） algorithm［J］. Information Sciences， 2017， 394： 38-52.
18	陈二静，姜恩波.文本相似度计算方法研究综述［J］.数据分析与知识发现，2017，1（6）：1-11.
	Chen E J， Jiang E B. Review of studies on text similarity measures［J］. Data Analysis and Knowledge Discovery，2017，1（6）：1-11.
19	Mikolov T， Sutskever I， Chen K， et al. Distributed representations of words and phrases and their compositionality［C］//Proceedings of the 26th International Conference on Neural Information Processing Systems， Lake Tahoe， Nevada， 5-10 December， Curran Associates Inc.， 2013： 3111-3119.
20	Devlin J， Chang M-W， Lee K， et al. BERT： Pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics， Minneapolis， MN， USA， 2-7 June， Association for Computational Linguistics， 2019： 4171-4186.
21	Vaswani A， Shazeer N， Parmar N， et al. Attention is all you need［C］//Proceedings of the 31st International Conference on Neural Information Processing Systems， Long Beach， California， USA， 4-9 December， Curran Associates Inc.， 2017： 6000-6010.
22	石磊，王毅，成颖，等.自然语言处理中的注意力机制研究综述［J］.数据分析与知识发现，2020，4（5）：1-14.
	Shi L， Wang Y， Cheng Y，et al. Review of attention mechanism in natural language processing［J］.Data Analysis and Knowledge Discovery，2020， 4（5）： 1-14.
23	冯兴杰，曾云泽.基于评分矩阵与评论文本的深度推荐模型［J］.计算机学报，2020，43（5）：884-900.
	Feng X J， Zeng Y Z. Joint deep modeling of rating matrix and reviews for recommendation［J］.Chinese Journal of Computers， 2020，43（5）：884-900.
24	杨国峰，杨勇.基于BERT的常见作物病害问答系统问句分类［J］.计算机应用，2020，40（6）：1580-1586.
	Yang G F， Yang Y. Question classification of common crop disease question answering system based on BERT［J］. Journal of Computer Applications， 2020，40（6）：1580-1586.
25	Giatsoglou M， Vozalis M G， Diamantaras K， et al. Sentiment analysis leveraging emotions and word embeddings［J］.Expert Systems with Applications， 2017， 69： 214-224.
26	Jha V， Savitha R， Shenoy P D， et al. A novel sentiment aware dictionary for multi-domain sentiment classification［J］.Computers & Electrical Engineering， 2018， 69： 585-597.
27	张公让，鲍超，王晓玉，等.基于评论数据的文本语义挖掘与情感分析［J］.情报科学，2021，39（5）：53-61.
	Zhang G R， Bao C， Wang X Y， et al. Sentiment analysis and text data mining based on reviewing data［J］. Information Science， 2021，39（5）：53-61.
28	崔雪莲，那日萨，刘晓君.基于主题相似性的在线评论情感分析［J］.系统管理学报，2018，27（5）：821-827.
	Cui X L， Na R S， Liu X J. Sentiment analysis of online reviews based on topic similarity［J］.Journal of Systems & Management， 2018，27（5）：821-827.
29	郑丽娟，王洪伟.基于情感本体的在线评论情感极性及强度分析：以手机为例［J］.管理工程学报，2017，31（2）：47-54.
	Zheng L J， Wang H W.Sentimental polarity and strength of online cellphone reviews based on sentiment ontology［J］.Jonrnal of Industrial Engineering/Engineering Management， 2017，31（2）：47-54.
30	Pang B， Lee L， Vaithyanathan S. Thumbs up？ sentiment classification using machine learning techniques［C］//Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing， USA， 2 July， Association for Computational Linguistics， 2002： 79-86.
31	Zubrinic K， Sjekavica T， Milicevic M， et al. A comparison of machine learning algorithms in opinion polarity classification of customer reviews［J］. International Journal of Computers， 2018， 20（3）： 159-163.
32	王立志，慕晓冬，刘宏岚.采用改进粒子群优化的SVM方法实现中文文本情感分类［J］.计算机科学，2020，47（1）：231-236.
	Wang L Z， Mu X D， Liu H L. Using SVM method optimized by improved particle swarm optimization to analyze emotion of Chinese text［J］.Computer Science， 2020，47（1）：231-236.
33	Xue J， Chen J， Hu R， et al. Twitter discussions and emotions about the COVID-19 pandemic： Machine learning approach［J］. J Med Internet Res， 2020， 22（11）： e20550.
34	王刚，李宁宁，杨善林.基于IDSSL的文本情感分析研究［J］.管理工程学报，2018，32（3）：126-133.
	Wang G， Li N N， Yang S L. Study of text sentiment analysis based on IDSSL［J］. Journal of Industrial Engineering/Engineering Management，2018，32（3）：126-133.
35	洪巍，李敏.文本情感分析方法研究综述［J］.计算机工程与科学，2019，41（4）：750-757.
	Hong W， Li M. A review： Text sentiment analysis methods［J］. Computer Engineering & Science， 2019，41（4）：750-757.
36	曾子明，万品玉.基于双层注意力和Bi-LSTM的公共安全事件微博情感分析［J］.情报科学，2019，37（6）：23-29.
	Zeng Z M， Wan P Y.Sentiment analysis of public safety events in micro-blog based on double-layeredattention and Bi-LSTM［J］. Information Science，2019，37（6）：23-29.
37	Khomsah S. Sentiment analysis on youtube comments using Word2Vec and random forest［J］. Telematika： Jurnal Informatika dan Teknologi Informasi， 2021， 18（1）： 61-72.
38	程艳，尧磊波，张光河，等.基于注意力机制的多通道CNN和BiGRU的文本情感倾向性分析［J］.计算机研究与发展，2020，57（12）：2583-2595.
	Cheng Y， Yao L B， Zhang G H， et al. Text sentiment orientation analysis of multi-channels CNN and BiGRU based on attention mechanism［J］. Journal of Computer Research and Development， 2020，57（12）：2583-2595.
39	徐绪堪，周泽聿.基于多尺度BiLSTM-CNN的微信推文的情感分类模型及应用研究［J］.情报科学，2021，39（5）：130-137.
	Xu X K， Zhou Z Y.A Multi-scale BILSTM-CNN based emotion classification model for WeChat tweets andits application［J］. Information Science， 2021，39（5）：130-137.
40	Wang P， Li J， Hou J. S2SAN： A sentence-to-sentence attention network for sentiment analysis of online reviews［J］. Decision Support Systems， 2021， 149： 113603.
41	国显达，那日萨，崔少泽.基于CNN-BiLSTM的消费者网络评论情感分析［J］.系统工程理论与实践，2020，40（3）：653-663.
	Guo X D， Na R S， Cui S Z. Consumer reviews sentiment analysis based on CNN-BiLSTM［J］. Systems Engineering-Theory & Practice， 2020，40（3）：653-663.
42	Bickerstaffe A， Zukerman I. A hierarchical classifier applied to multi-way sentiment detection［C］//Proceedings of the 23rd International Conference on Computational Linguistics， Beijing， China， 23-27 August， Association for Computational Linguistics， 2010： 62-70.
43	Robertson S. Understanding inverse document frequency： On theoretical arguments for IDF［J］. Journal of Documentation， 2004， 60（5）： 503-520.
44	Hinton G E. Learning distributed representations of concepts［C］//Proceedings of the Eighth Annual Conference of the Cognitive Science Society， Amherst， Massachusetts， August 15-17， 1986.
45	Pouransari H， Ghili S. Deep learning for sentiment analysis of movie reviews［R］.Working Paper， CS224N Project of Stanford University， 2014.
46	唐明，朱磊，邹显春.基于Word2Vec的一种文档向量表示［J］.计算机科学，2016，43（6）：214-217+269.
	Tang M， Zhu L， Zou X C. Document vector representation based on Word2Vec［J］. Computer Science， 2016，43（6）：214-217+269.
47	Garrido-Merchan E C， Gozalo-Brizuela R， Gonzalez-Carvajal S. Comparing BERT against traditional machine learning models in text classification［J］. Journal of Computational and Cognitive Engineering， 2023， 2（4）： 352-356.
48	胡春涛，秦锦康，陈静梅，等.基于BERT模型的舆情分类应用研究［J］.网络安全技术与应用，2019（11）：41-44.
	Hu C T， Qin J K， Chen J M， et al. Application research of public opinion classification based on BERT model［J］. Network Security Technology & Application， 2019（11）：41-44.
49	Chiorrini A， Diamantini C， Mircoli A， et al. Emotion and sentiment analysis of tweets using BERT［C］//Proceedings of the EDBT/ICDT 2021 Joint Conference Workshops， Nicosia， Cyprus， March 23， 2021.
50	Lou G， Shi H. Face image recognition based on convolutional neural network［J］. China Communications， 2020， 17（2）： 117-124.
51	Fernandez C S， Martin E， Perucho S， et al. High interpretable machine learning classifier for early glaucoma diagnosis［J］. International Journal of Ophthalmology， 2021， 14（3）： 393-398.
52	Manek A S， Shenoy P D， Mohan M C， et al. Aspect term extraction for sentiment analysis in large movie reviews using gini index feature selection method and SVM classifier［J］. World Wide Web， 2017， 20（2）： 135-154.
53	Troussas C， Virvou M， Espinosa K J， et al. Sentiment analysis of facebook statuses using naive bayes classifier for language learning［C］//IISA 2013. Piraeus， Greece， July 10-12， IEEE， 2013： 1-6.
54	Rathi M， Malik A， Varshney D， et al. Sentiment analysis of tweets using machine learning approach［C］//2018 Eleventh International Conference on Contemporary Computing （IC3）.Noida，India，Aug 2-4， IEEE， 2018： 1-3.
55	Woloszynski T， Kurzynski M， Podsiadlo P， et al. A measure of competence based on random classification for dynamic ensemble selection［J］.Information Fusion， 2012， 13（3）： 207-213.
56	Galar M， Fernández A， Barrenechea E， et al. DRCW-OVO： Distance-based relative competence weighting combination for one-vs-one strategy in multi-class problems［J］．Pattern Recognition， 2015， 48（ 1）： 28-42.
57	Friedman J H. Another approach to polychotomous classification［R］. Working Paper， Department of Statistics of Stanford University， 1996.
58	Hüllermeier E， Vanderlooy S. Combining predictions in pairwise classification： An optimal adaptive voting strategy and its relation to weighted voting［J］. Pattern Recognition， 2010， 43（1）： 128-142.
59	Zhou Z H. Ensemble methods： Foundations and algorithms［M］. London： Chapman and Hall/CRC， 2012.
60	Rokach L. Data mining and knowledge discovery handbook［M］. Boston： Springer， 2005.
61	Cruz R M O， Sabourin R， Cavalcanti G D C. Dynamic classifier selection： Recent advances and perspectives［J］. Information Fusion， 2018， 41： 195-216.
62	Mendialdua I， Martinez J M， Rodriguez I， et al. Dynamic selection of the best base classifier in one versus one［J］. Knowledge-Based Systems， 2015， 85： 298-306.
63	Pang B， Lee L. Seeing stars： Exploiting class relationships for sentiment categorization with respect to rating scales［C］//Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics， Ann Arbor， Michigan， June 25-30 ，Association for Computational Linguistics，2005： 115-124.
64	Wu Y， Schuster M， Chen Z， et al. Google's neural machine translation system： Bridging the gap between human and machine translation［J］. CoRR， 2016， abs/1609： 08144.

数据集	评论条数	类别数
D1	1027	3
D2	1027	4
D3	902	3
D4	902	4
D5	1307	3
D6	1307	4
D7	1770	3
D8	1770	4

数据集	TF-IDF		Word2Vec		BERT
数据集	Acc	Kappa	Acc	Kappa	Acc	Kappa
D1	42.71	0.0297	51.02	0.1834	65.43	0.4576
D2	43.69	0.0167	43.75	0.0181	57.12	0.3538
D3	43.82	0.1020	41.94	0.0685	62.31	0.4260
D4	36.62	0.0030	39.47	0.0567	54.33	0.3350
D5	51.11	0.0659	50.27	0.0506	78.81	0.6470
D6	45.60	0.0000	45.60	0.0000	67.43	0.5129
D7	52.20	0.2224	48.32	0.1522	72.49	0.5751
D8	42.61	0.0277	44.52	0.0247	65.60	0.4844
平均值	44.80	0.0584	45.61	0.0692	65.44	0.4739

数据集	DYN		DCS		DES
数据集	Acc	Kappa	Acc	Kappa	Acc	Kappa
D1	63.39	0.4338	59.52	0.3439	65.43	0.4576
D2	54.43	0.3291	52.91	0.2814	57.12	0.3538
D3	59.76	0.3871	56.89	0.3340	62.31	0.4260
D4	52.52	0.3229	47.26	0.2226	54.33	0.3350
D5	76.59	0.6137	74.76	0.5728	78.81	0.6470
D6	64.27	0.4763	62.85	0.4380	67.43	0.5129
D7	70.40	0.5447	67.34	0.4958	72.49	0.5751
D8	62.82	0.4566	61.36	0.4101	65.60	0.4844
平均值	63.02	0.4455	60.361	0.3873	65.44	0.4739

数据集	Pang等^[63]		Liu等^[17]		Bickerstaffe等^[42]		BERT+DES
数据集	Acc	Kappa	Acc	Kappa	Acc	Kappa	Acc	Kappa
D1	59.95	0.3756	62.48	0.4268	59.30	0.3393	65.43	0.4576
D2	48.23	0.2257	51.97	0.2876	53.62	0.3230	57.12	0.3538
D3	58.24	0.3619	56.35	0.3364	54.88	0.3141	62.31	0.4260
D4	49.56	0.2681	46.64	0.2412	48.04	0.2348	54.33	0.3350
D5	73.89	0.5608	72.71	0.5479	71.75	0.5302	78.81	0.6470
D6	60.64	0.4084	59.78	0.4075	62.84	0.4518	67.43	0.5129
D7	67.65	0.5011	72.00	0.5717	67.46	0.4988	72.49	0.5751
D8	57.72	0.3658	63.47	0.4678	60.92	0.4254	65.60	0.4844
平均值	59.49	0.3834	60.68	0.4109	59.85	0.3896	65.44	0.4739

[1]	SHI Yuan, WANG Xin-hua, GAO Hong-wei. Pricing Strategy and Optimal Shareholding of Bertrand Duopoly Firms with Cross-shareholding [J]. Chinese Journal of Management Science, 2021, 29(2): 42-50.
[2]	PU Yong-jian, SUN Xian-hua. A Bertrand Model based on Contracts for Difference for Inhibiting Market Power [J]. Chinese Journal of Management Science, 2017, 25(5): 109-115.
[3]	JI Shou-feng, JIANG Li-wen, SUN Qi, YU Hai-fei. Research onNash Equilibrium Strategies by Considering Brand Effect in Multi-Products' and Multi-Manufacturers' Supply Chain [J]. Chinese Journal of Management Science, 2017, 25(5): 97-108.
[4]	ZHOU Xiong-Wei, LIU Peng-chao, CHEN Xiao-hong. Quality Differentiation Productfalse Information Research in the Duopoly Market Under the Conditions of Asymmetric Information [J]. Chinese Journal of Management Science, 2016, 24(3): 133-140.
[5]	ZHANG Wei, ZHONG Wei-jun, MEI Shu-e. Study on Foreign Penetration, Privatization Degree and Social Welfare with Product Differentiation [J]. Chinese Journal of Management Science, 2016, 24(11): 11-18.
[6]	JIN Zhi-Hong, QIU Bo. The Coordinated Optimization on Berthing Containerships between the Dock and Pontoon in a Regional Hub Port [J]. Chinese Journal of Management Science, 2011, 19(4): 105-110.
[7]	FANG Hai-yan, DA Qing-li . Research of Optimal R&D Subsidy Strategic with Product Differentiation [J]. Chinese Journal of Management Science, 2009, 17(3): 166-172.
[8]	YANG Xiao-hua, LUO Yun-feng, WU Hui-qiu. Bertrand Model and the Theory of Supermodular Games [J]. Chinese Journal of Management Science, 2009, 17(1): 95-100.
[9]	GUO Xue-song, YUAN Zhi-ping, LIU Bo. Study on Multi-class Classification Method Based on Semi-fuzzy Hypersphere Support Vector Machine [J]. Chinese Journal of Management Science, 2008, 16(2): 140-144.
[10]	YANG Guang-qing, JIANG Lu-quan, WANG Huan-chen, CHEN Ming-yi. The Game Analysis on Financing Strategy and Product Differentiation Strategy in the Frame of Bertrand Competition [J]. Chinese Journal of Management Science, 2006, (4): 88-94.
[11]	CHEN Bo, ZHOU Ya-ping, HUA Zhong-sheng. Analysis and Simulation of Internet Pricing Based on the Information of Customer Consumption [J]. Chinese Journal of Management Science, 2005, (4): 111-114.

Researchon Multi-class Sentiment Classification Based on BERT and Dynamic Ensemble Selection

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 64

Related Articles 11

Metrics

Comments

Recommended 0