Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation

Abstract

One-on-one tutoring is an effective instructional method for enhancing learning, yet its efficacy hinges on tutor competencies. Novice math tutors often prioritize content-specific guidance, neglecting aspects such as social-emotional learning. Social-emotional learning promotes equity and inclusion and nurtures relationships with students, which is crucial for holistic student development. Assessing the competencies of tutors accurately and efficiently can drive the development of tailored tutor training programs. However, evaluating novice tutor ability during real-time tutoring remains challenging as it typically requires experts-in-the-loop. To address this challenge, this study harnesses Generative Pre-trained Transformers (GPT), such as GPT-3.5 and GPT-4, to automatically assess tutors’ ability of using social-emotional tutoring strategies. Moreover, this study also reports on the financial dimensions and considerations of employing these models in real-time and at scale for automated assessment. Four prompting strategies were assessed: two basic Zero-shot prompt strategies, Tree of Thought prompting, and Retrieval-Augmented Generator (RAG) prompting. The results indicate that RAG prompting demonstrated the most accurate performance (assessed by the level of hallucination and correctness in the generated assessment texts) and the lowest financial costs. These findings inform the development of personalized tutor training interventions to enhance the the educational effectiveness of tutored learning.