Assessing EAP course effectiveness via learner corpus analysis

Abstract

Despite English being the dominant language in HK tertiary education, expectations regarding the effectiveness of initial EAP programs are ‘unrealistic’ (Bruce & Hamp-Lyons, 2015). EAP course effectiveness is typically measured via the triangulation of test scores and student-led post-course evaluations of course and teacher performance, which are [unfortunately] both summative and subjective in nature and do not generally take into account assessment of the actual linguistic data that was produced over time, tracking student production from pre- to post- training. Such an investigation would constitute a new 'affordance' among the 'multiple affordances' (Leńko-Szymańska & Boulton, 2015) of language corpora currently available to educators, and would be particularly useful if course teachers were involved in the corpus design (e.g. Urzúa, 2015). The methodology involved the construction, annotation and analysis of a pilot learner corpora according to the principles of Granger's (2015) Contrastive Interlanguage Analysis (CIA2), where 350,000 words were taken longitudinally (pre-, mid- and post-EAP training) from written academic essays and reports. The corpus' annotation scheme was derived from EAP course content alongside consultation with course teachers, and includes annotation for over 20 error categories, the use of interpersonal and intertextual metadiscursive language (following Hyland, 2000), and the use of in-text citation / reference lists. Post-hoc analysis also includes multidimensional analysis (e.g. Biber, 1988) using the MAT program, (Nini, 2014). Corpus building and annotation was performed using UAMCorpustool (O'Donnell, 2008). Results There is little impact of instruction on common L2 errors found between pre- and post-training data in most (but not all) categories. Variation in the use of key stance features including hedging, boosting, epistemic nouns, attitude markers and self-mentions suggests our L2 writers are able to produce more nuanced positions on an academic topic. The results of the multidimensional analysis exhibit considerable variation between data points in the direction of academic discourse across all five dimensions analysed, including a drop in the use of first person pronouns and the mechanical use of discourse connectives, alongside an increased emphasis on nominalisation. The results provide one of the first quantitative, corpus-driven measures of EAP course effectiveness. The findings suggest a warmly positive effect of EAP instruction on learner production after only a single semester. A number of pedagogical opportunities for the data are also outlined including the benefits of such analysis for written corrective feedback and future analysis of discipline-specific L2 discourse. The results will hopefully lead to improved data-driven course provision (e.g. Johns, 1991), and as such are likely to be of great importance for ELT course administrators and teachers.