Regression Towards the Mean Artifacts and Matthew Effects in multilevel analyses of value-added of individual schools

Abstract

League tables are a problematic approach to inferring school effectiveness, but traditional value-added approaches are fraught with statistical complexities. According to the Regression Towards the Mean Artifacts (RTMA), students with initially high or low scores tend to regress towards the mean in subsequent testing, resulting in biased estimates of school growth (Marsh & Hau, 2002). The Matthews Effect is an apparently counter-balancing artifact in growth in achievement gains is systematically larger for students who are initially more able. (i.e., the rich becomes richer). Mathematical proof shows that although the Matthew and the RTMA artifacts work in opposite direction and tend to cancel each other, they share a similar mechanism and can be rectified. In this study, mathematical derivations and Monte Carlo simulated data are used to compare four models, namely: (i) without any remedy, (ii) with remedy for Matthew effect only, (iii) with remedy for RTMA only, (iv) remedies for both Matthew and RTMA effects. The conditional strategy with individual assignment test scores (used in assigning students to different schools) as covariate remedies artifacts, consistent with Marsh & Hau's (2002) conclusion for RTMA. The associated problems with the two effects in estimating school value-added information are discussed.