The Wald test for empirical Q-matrix validation

Abstract

The Q-matrix that specifies required attributes for each item is a crucial component of cognitive diagnostic assessments (CDAs). However, conventional Q-matrix development involves some chance of subjectivity, which can result in validity concerns. The validity inferences from CDAs can be improved by statistical analyses. This study proposes a new procedure, Wald-Q, which is the adaptation of a statistical test – the Wald test. The Wald-Q is a multivariate hypothesis testing that simultaneously compares all possible qvectors at the item level. A simulation study is carried out under varying conditions (i.e., sample sizes, test lengths, number of random misspecifications, and item qualities) to examine the viability of the Wald-Q. In addition, situations in which the true underlying restricted model is known and unknown are considered. Findings are reported as the proportions correct specifications retained and misspecifications corrected in the Q-matrix. Based on the preliminary results, when there are no misspecifications, the Wald-Q perfectly retains all the correct specifications for the item. When misspecifications are present, the retention rates for correct specifications are generally perfect with some exceptions under low quality items, short test lengths and small sample sizes; in addition, the correction rate for misspecifications is excellent, except when the item quality is low. Finally, the procedure is compared to an existing Q-matrix validation procedure, and is shown to provide better results.