The effect of confidence expressed by AI on students' agreement with ChatGPT responses

Abstract

Generative artificial intelligence (AI) is used in higher education as a tool to support academic writing, learning, and decision-making. This thesis examines whether confidence expressed by ChatGPT influences students' agreement with those responses. A mixed-methodology approach includes a structured literature review and a quantitative experimental survey. Students from Finnish higher education institutions rated AI-generated responses, presented with three confidence levels using a five-point Likert scale. Survey data were analysed in Python using descriptive statistics, the Friedman test, the Wilcoxon signed-rank test with Bonferroni correction, and repeated-measures analysis of variance (ANOVA). The results showed a statistically significant difference in agreement between confidence conditions. Responses expressing high confidence received the highest mean agreement scores, while those expressing neutral confidence received the lowest. Pairwise comparisons showed that agreement with responses expressing high confidence was significantly higher than with responses expressing neutral confidence, after adjusting for multiple comparisons. However, the observed effect size was small. The results highlight the importance of promoting the responsible design and use of AI systems.