Do I complete Q?

Abstract

A central component for most cognitive diagnosis models (CDMs) is an item and attribute association matrix (Q‐matrix; Tatsuoka, 1983), which specifies whether an attribute is measured by each item. A complete Q‐matrix, which may or may not involve an identity matrix, is necessary for the identification of all attribute profiles. However, the completeness, or lack thereof, of a particular Q‐matrix may vary from one CDM to another. A method that has been proposed by Kohn & Chiu (2017) to assess Q‐matrix completeness is to compare the success probabilities across the items of the different attribute profiles. This method presupposes that the underlying CDMs are known, a condition that is difficult to meet in practice. The current work proposes a simulation‐based approach to assess Q‐matrix completeness. The proposed method involves determining the simplest CDMs empirically, and disentangling completeness from test reliability. A simulation study is conducted to evaluate the viability of the proposed method. Results show that the simulation‐based method performs well under most conditions, but needs to be used with caution when the sample size is small and items are of inadequate quality. A set of real data is also analyzed to examine the proposed procedure.