Alternative implementations of the GDI Q-matrix validation procedure

Abstract

Misspecification of the Q-matrix under Cognitive Diagnostic Models (CDM) can affect correct classification of examinees. Previous researchers have developed several methods to validate a Q-matrix based on special CDMs (Templin & Henson, 2006; DeCarlo, 2012; Chiu & Douglas, 2013). De la Torre and Chiu (2016) proposed a discrimination index (sigma squared) that can be used to identify and replace misspecified Q-matrix entries using a general CDM (i.e., the Generalized Deterministic Input, Noisy, 'And' gate [G-DINA] model) and applicable to the reduced models it subsumes. However, a cutoff ε for the Proportion of Variance Accounted For (PVAF) by a particular q-vector relative to the maximum sigma squared needs to be predetermined. It was set to be 0.95 in the study (de la Torre & Chiu, 2016), without further justification. Despite promising results, choosing the best cutoff value can in practice be difficult. This study proposes two methods to validate a Q-matrix based on the PVAF using the G-DINA model, but without the need to specify ε a priori. Method 1 selects the best q-vector based on the 'mesa' plot, which shows the PVAF values against the number of attributes specified; Method 2 selects the best candidate q-vector based on their AIC and BIC indices. The proposed methods, together with the current implementation, will be compared in terms of their q-vector recovery rates. The factors considered in the simulation study includes the number of attributes (K), the number of items (J), sample size (N), and the pattern of Q-matrix misspecifications.