Individualized feedback to raters : effects on rating severity, inconsistency, and bias in the context of Chinese as a second language writing assessment

Abstract

Performance-based language assessment commonly requires human raters to assign scores to language leaners’ performances. The subjectivity in human ratings inevitably introduces rater variability that has been identified as a main source of construct-irrelevant variance. Individualized feedback has been used in rater training to limit such variance and increase the reliability of performance-based language assessment. This training method, however, has not yet proved to be effective in improving raters’ rating outcomes (i.e., severity, inconsistency, and bias towards a rating scale category). Moreover, few previous studies have investigated the effects of feedback frequency at a given time period on raters’ rating outcomes. The present study examined the immediate and retention effects of individualized feedback on raters’ rating outcomes, as well as the effects of raters’ perceptions of the individualized feedback on rating outcomes. The participants were 93 native Chinese speakers without previous rating experience, and they were randomly assigned to one of three treatment groups. The three groups differed in the way of receiving individualized feedback at a given time period: (a) control group receiving no feedback, (b) single-feedback group receiving the feedback once, and (c) double-feedback group receiving the feedback twice. Each participant rated 100 writing scripts on Day 1 as the pre-feedback ratings, and received one of the feedback treatments on Day 2. The post-feedback ratings were conducted immediately after the feedback session on Day 2 by assigning each participant 100 new writing scripts to rate. This was followed by a questionnaire and interview for the double-feedback and single-feedback groups on the same day. Raters’ retention of the feedback was measured by assigning each participant 100 new writing scripts to rate as the delayed post-feedback ratings after one week. Raters’ rating outcomes were yielded through multi-faceted Rasch modeling. One-way ANCOVAs, repeated measures ANOVAs, Kruskal-Wallis H tests, and two-way ANCOVAs were run to test the hypotheses. Based on the results of this study, the following main conclusions were made. First, individualized feedback significantly affected raters’ rating severity and inconsistency. The rating severities of the double-feedback and single-feedback groups were superior to the control group. In addition, the rating inconsistency of the double-feedback group was superior to the single-feedback group. Second, with regard to the retention effects, individualized feedback was found to be beneficial to raters from the double-feedback group to retain their improvements in rating severity. It also helped raters from the single-feedback group to retain their improvements in rating inconsistency. Third, in terms of the effects of raters’ perceptions of the individualized feedback, the results revealed that raters’ perceptions of usefulness of the individualized feedback affected rating severity, inconsistency, and bias towards coherence. Furthermore, raters’ perceptions of recall of the individualized feedback during subsequent ratings affected rating inconsistency. In addition, raters’ perceptions of incorporation of the individualized feedback into subsequent ratings affected rating severity. These findings may shed light on the application of individualized feedback in the designs of face-to-face and online training programs.