Current Limitations in Cyberbullying Detection: on Evaluation Criteria, Reproducibility, and Data Scarcity
Emmery,Chris Verhoeven,Ben De Pauw,Guy Jacobs,Gilles Van Hee,Cynthia Lefever,Els Desmet,Bart Hoste,Véronique Daelemans,Walter
Emmery,Chris
Verhoeven,Ben
De Pauw,Guy
Jacobs,Gilles
Van Hee,Cynthia
Lefever,Els
Desmet,Bart
Hoste,Véronique
Daelemans,Walter
Abstract
The detection of online cyberbullying has seen an increase in societal importance, popularity in research, and available open data. Nevertheless, while computational power and affordability of resources continue to increase, the access restrictions on high-quality data limit the applicability of state-of-the-art techniques. Consequently, much of the recent research uses small, heterogeneous datasets, without a thorough evaluation of applicability. In this paper, we further illustrate these issues, as we (i) evaluate many publicly available resources for this task and demonstrate difficulties with data collection. These predominantly yield small datasets that fail to capture the required complex social dynamics and impede direct comparison of progress. We (ii) conduct an extensive set of experiments that indicate a general lack of cross-domain generalization of classifiers trained on these sources, and openly provide this framework to replicate and extend our evaluation criteria. Finally, we (iii) present an effective crowdsourcing method: simulating real-life bullying scenarios in a lab setting generates plausible data that can be effectively used to enrich real data. This largely circumvents the restrictions on data that can be collected, and increases classifier performance. We believe these contributions can aid in improving the empirical practices of future research in the field.
Description
Funding Information: The work presented in this article was carried out in the framework of the AMiCA (IWT SBO-project 120007) project, funded by the government agency for Innovation by Science and Technology (IWT). Publisher Copyright: © 2020, The Author(s).
Date
2021
Journal Title
Journal ISSN
Volume Title
Publisher
Research Projects
Organizational Units
Journal Issue
Keywords
cyberbullying detection, cybersecurity, machine learning, benchmarking, resource evaluation, cross-domain, reproducibility, crowdsourcing, SDG 16 - Peace, Justice and Strong Institutions
Citation
Emmery, C, Verhoeven, B, De Pauw, G, Jacobs, G, Van Hee, C, Lefever, E, Desmet, B, Hoste, V & Daelemans, W 2021, 'Current Limitations in Cyberbullying Detection : on Evaluation Criteria, Reproducibility, and Data Scarcity', Language Resources and Evaluation, vol. 55, pp. 597-633. https://doi.org/10.1007/s10579-020-09509-1