Are perfect transcripts necessary when we analyze classroom dialogue using AIoT?

Abstract

<p>Classroom dialogue plays a crucial role in enhancing the quality of teaching and learning. Many researchers have utilized artificial intelligence (AI) and Internet of things (IoT) to develop models and systems for automatic analysis and feedback. However, the question of whether we should employ these AIoT tools on automatic transcripts of classroom dialogue generated by automatic speech recognition software or on transcripts that have undergone human revision remains unresolved, which involves the trade-off between accuracy and efficiency. Thus, this paper examines whether perfect transcripts are needed to analyze talk moves in classroom dialogue. We initially constructed two deep learning models to analyze teacher talk moves in K-12 mathematics lessons. Subsequently, we collected an additional set of six K-12 mathematics lesson videos and used a classroom dialogue analysis system equipped with speech recognition software to automatically transcribe them, resulting in ASR_pure transcripts. These transcripts were then manually revised and verified to create ASR_human transcripts. A comparison between the two types of transcripts revealed evident errors in the ASR_pure transcripts. Next, we employed the developed AI models to predict talk moves in both ASR_pure and ASR_human transcripts and assessed their consistency. The findings demonstrate a high level of consistency in talk move prediction between the two types of transcripts across the six lessons. Furthermore, the ASR_pure transcripts also exhibit high consistency in specific talk moves (e.g., pressing for accuracy) when compared to ASR_human transcripts. We propose a hypothesis that this consistency between inaccurate ASR_pure transcripts and perfect ASR_human transcripts can be attributed to the ASR software accurately recognizing key indicators that serve as talk moves, rather than accurately identifying every word. Notably, upon removing the key indicator words from teacher utterances, both the talk move level and lesson level consistency experience a substantial decline. Therefore, we suggest that perfect transcripts of classroom dialogue may not be necessary for AIoT to analyze teacher talk moves and provide teachers with prompt and accurate feedback, especially when the ASR software can accurately recognize keywords.</p>