Improving the quality of teaching by utilising written student feedback: A streamlined process

Abstract

Currently student feedback is mainly evaluated with quantitative methods since qualitative analysis has been highly effort intensive. In this article, we present a process for tapping into the resource of responses to open-ended feedback questions by using a topic-modelling approach that goes beyond listing modelling outcomes. The objective of this study is to present a streamlined, yet rigorous, process for analysing large amounts of written feedback that connects qualitative findings to existing literature, theories and quantitative feedback. The topic models are created using the Latent Dirichlet Allocation (LDA) method, after which qualitative and quantitative evaluation methods are used to validate the topic outcomes. The proposed process can help educators analyse teaching quality on programme- or institution-wide level, or on single courses with a very large number of students. The process systematizes and combines existing processes, is repeatable, and can serve as a basis for richer analysis for educators. In student evaluation of teaching (SET) research, it advances the state of the art in applied topic modelling by demonstrating how to validate the topics via thematic analysis and by connecting them to theoretical frameworks and quantitative data. Previous topic modelling studies in this field follow mainly descriptive approaches. We demonstrate the process with feedback data collected from 6087 student evaluations of university courses and confirm that quantitative feedback variables can be used to validate qualitative feedback topic-modelling outcomes and thematic analysis provides a more in-depth explanation of the topics. We additionally find that the proposed topic modelling approach discovers new constructs of SET that cannot be distinguished from quantitative SET measures. The main limitation of the study is that the proposed process is novel and requires further evaluation to establish its full validity.