Evaluating multidimensional extensions of the Elo rating systems for tracking ability in online learning environments
Vremeiren,Hanke Hofman,Abe D. Bolsinova,Maria
Vremeiren,Hanke
Hofman,Abe D.
Bolsinova,Maria
Abstract
The traditional Elo rating system (ERS), widely used as a student model in adaptive learning systems, assumes unidimensionality (i.e., all items measure a single ability or skill), limiting its ability to handle multidimensional data common in educational contexts. In response, several multidimensional extensions of the Elo rating system have been proposed, yet their measurement properties remain underexplored. This paper presents a comparative analysis of two such multidimensional extensions specifically designed to address within-item dimensionality: the multidimensional extension of the ERS (MERS) by Park et al. (2019) and the Multi-Concept Multivariate Elo-based Learner model (MELO) introduced by Abdi et al. (2019). While both these systems assume a compensatory multidimensional item response theory model underlying student responses, they propose different ways of updating the model parameters. We evaluate these algorithms in a simulation study using key performance metrics, including prediction accuracy, speed of convergence, bias, and variance of the ratings. Our results demonstrate that both multidimensional extensions outperform the unidimensional Elo rating system when the underlying data is multidimensional, highlighting the importance of considering multidimensional approaches to better capture the complexities inherent to the data. Furthermore, our results demonstrate that while the MELO algorithm is converging faster, it exhibits significant bias and lower prediction accuracy compared to the MERS. In addition, the MERS's robustness to misspecifications of the Q-matrix and its weights gives it an edge in situations where generating an accurate Q-matrix is challenging.
Description
Date
2025-07
Journal Title
Journal ISSN
Volume Title
Publisher
International Educational Data Mining Society
Research Projects
Organizational Units
Journal Issue
Keywords
Multidimensionality, Student modeling, Elo Rating System, Online Education
Citation
Vremeiren, H, Hofman, A D & Bolsinova, M 2025, Evaluating multidimensional extensions of the Elo rating systems for tracking ability in online learning environments. in C Mills, G Alexandron, D Taibi, G Lo Bosco & L Paquette (eds), Proceedings of the 18th international conference on educational data mining. International Educational Data Mining Society, pp. 143-154, Educational Data Mining Conference 2025 , Palermo, Italy, 20/07/25. https://doi.org/10.5281/zenodo.15870211