Performance of the Q-Matrix Validation Methods in the DINA Model

Abstract

All cognitive diagnostic models that evaluate educational test data require a Q-matrix that combines every item in a test with the required cognitive skills for each item to be answered correctly. Generally, the Q-matrix is constructed by education experts' judgment, leading to some uncertainty in its elements. Various statistical methods are suggested to validate misspecifications in the Q-matrix. This paper evaluates the performance of the Q-matrix validation methods, thesequential expectation-maximization-based δ-method (SEM δ-method), and the Q-matrix refinement (QMR) method using a study with real data and simulations.The simulation design results showed that the misspecification percentage and the length of the test had a small or no effect on the mean q-entry recovery rates (MRRs) of both methods, while the increase in sample size had an improving effect. The MRRs of both methods decreased when the number of attributes and guessing (g) -slip (s) parameters increased. According to simulation study results, the QMR method performed better than the SEM δ-method. For the q-matrix validation, it can be suggested that CDM practitioners prioritize the QMR method and use a sample size of 1,000. On the other hand, the real data results revealed that the MRRs of both methods were at the base rates. This result highlights the need for further research on method comparison, specifically for real-world data applications where the number of attributes is relatively large and the test duration is short.