Confirmation bias between human software testers and Claude 3.7 Sonnet : quantitative and qualitative insights

Abstract

Confirmation bias is a tendency of human reasoning that leads people to seek and favour evidence supporting their existing beliefs. In software testing, it manifests as a tendency to confirm that software behaves as specified rather than to test the defects that threaten its quality. As large language models (LLMs) are increasingly used to generate and support testing, it is not yet known whether they reproduce this same bias. This thesis investigated whether Claude (Claude 3.7 Sonnet), as a representative LLM, exhibits confirmation bias when generating functional (black box) test cases, and how its output differs from that of human testers. Two research questions were addressed: whether confirmation bias manifestation differs between Claude-generated test cases and human-designed ones, and what qualitative patterns distinguish between the two.

The study replicated the controlled experiments by Salman et al. (2019, 2023). Twenty-five independent Claude instances generated functional test cases using the MusicFone requirements specification, while the human data was reused from the controlled experiments by Salman et al. Confirmation bias was measured through a z-metric for humans and Claude, and complemented by a qualitative analysis of the generated test cases. The results showed that there’s a statistically significant difference between confirmation bias of Claude and human testers. Descriptive statistics revealed; Claude instances leaned toward more confirming specified behaviour. The qualitative findings showed that Claude's test cases were more formulaic, repetitive, and narrowly focused, whereas the human-designed cases were more varied and exploratory. These findings extend confirmation-bias research from human testers to LLMs in software testing and indicate that Claude while can support functional test-case generation, it also exhibits testing confirmation bias. Hence, its output should not be assumed to be neutral bias-free and requires a human review.