Do Many Models Make Light Work? Evaluating Ensemble Solutions for Improved Rumor Detection

Younghwan Kim, Huy Kang Kim, Hyoungshick Kim, Jin B. Hong

Research output: Contribution to journalArticlepeer-review

10 Citations (Scopus)

Abstract

There have been many efforts to detect rumors using various machine learning (ML) models, but there is still a lack of understanding of their performance against different rumor topics and available features, resulting in a significant performance degrade against completely new and unseen (unknown) rumors. To address this issue, we investigate the relationship between ML models, features, and rumor topics to select the best rumor detection model under specific conditions using 13 different ML models. Our experiment results demonstrate that there is no clear winner among the ML models in all different rumor topics with respect to the detection performance. To overcome this problem, a possible way is to use an ensemble of ML models. Although previous work presents an improved detection of rumors using ensemble solutions (ES), their evaluation did not consider detecting unknown rumors. Further, they did not present nor evaluate the configuration of the ES to ensure that it indeed performs better than using a single ML model. Based on these observations, we propose to evaluate the use of an ES by examining their unknown rumor detection performance compared with single ML models but as well as different configurations of the ESes. Our experimental results using real-world datasets found that an ES of Random Forest, XGBoost and Multilayer perceptron overall produced the best F1 score of 0.79 for detecting unknown rumors, a significant improvement compared with a single best ML model which only achieved a 0.58 F1 score. We also showed that not all ESes are the same, with significantly degraded detection and large variations in performance when different ML models are used to construct the ES. Hence, it is infeasible to rely on any single ML model-based rumor detector. Finally, our solution also performed better than other recent detectors, such as eventAI and NileTMRG that performed similar to using a single ML model - making it a much more attractive solution to detect unknown rumors in practice.

Original languageEnglish
Article number9167196
Pages (from-to)150709-150724
Number of pages16
JournalIEEE Access
Volume8
DOIs
Publication statusPublished - 2020

Bibliographical note

Funding Information:
This work was supported in part by the Australia-Korea Foundation (AKF) of the Department of Foreign Affairs and Trade (DFAT) project Intelligent Cybersecurity: Jumpstarting Collaboration on Smarter Security for Dynamic Networks, and in part by the framework of international cooperation program managed by the National Research Foundation of Korea under Grant NRF-2017K1A3A1A17092614.

Publisher Copyright:
© 2013 IEEE.

Keywords

  • Feature analysis
  • Twitter
  • ensemble solution
  • machine learning
  • rumor detection
  • social media

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering

Fingerprint

Dive into the research topics of 'Do Many Models Make Light Work? Evaluating Ensemble Solutions for Improved Rumor Detection'. Together they form a unique fingerprint.

Cite this