Sensitivity of the RMSD for detecting item-level misfit in low-performing countries
Tijmstra,Jesper Bolsinova,Maria Liaw,Yuan-Ling Rutkowski,Leslie Rutkowski,David
Tijmstra,Jesper
Bolsinova,Maria
Liaw,Yuan-Ling
Rutkowski,Leslie
Rutkowski,David
Abstract
Although the root-mean squared deviation (RMSD) is a popular statistical measure for evaluating country-specific item-level misfit (i.e., differential item functioning [DIF]) in international large-scale assessment, this paper shows that its sensitivity to detect misfit may depend strongly on the proficiency distribution of the considered countries. Specifically, items for which most respondents in a country have a very low (or high) probability of providing a correct answer will rarely be flagged by the RMSD as showing misfit, even if very strong DIF is present. With many international large-scale assessment initiatives moving toward covering a more heterogeneous group of countries, this raises issues for the ability of the RMSD to detect item-level misfit, especially in low-performing countries that are not well-aligned with the overall difficulty level of the test. This may put one at risk of incorrectly assuming measurement invariance to hold, and may also inflate estimated between-country difference in proficiency. The degree to which the RMSD is able to detect DIF in low-performing countries is studied using both an empirical example from PISA 2015 and a simulation study.
Description
Funding Norges Forskningsråd. Grant Number: 255246
Date
2020
Journal Title
Journal ISSN
Volume Title
Publisher
Research Projects
Organizational Units
Journal Issue
Keywords
Citation
Tijmstra, J, Bolsinova, M, Liaw, Y-L, Rutkowski, L & Rutkowski, D 2020, 'Sensitivity of the RMSD for detecting item-level misfit in low-performing countries', Journal of Educational Measurement, vol. 57, no. 4, pp. 566-583. https://doi.org/10.1111/jedm.12263