Automated testing : bug identification using large language models
Poudel, Aashik (2026)
Kandidaatintyö
2026
School of Engineering Science, Tietotekniikka
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe20260621100684
https://urn.fi/URN:NBN:fi-fe20260621100684
Tiivistelmä
This thesis investigates the role Large Language Models (LLMs) play within automated software testing workflows, focusing specifically on their capacity to detect software bugs. Given that software applications are increasingly becoming complex, there is a need to find smart ways to carry out software testing in order to identify bugs which cannot be detected through conventional means. To achieve this, this study adopts a mixed methodology involving systematic literature review, a web-based survey of 18 software experts and development of a software bug detection system called BugLens.
The literature review focuses on the evaluation of LLMs for tasks including test case generation, bug discovery, vulnerability discovery, test oracle generation and automated software fixing. The online survey collects the opinions of practitioners on the use, reliability and readiness of using LLMs for testing purposes. BugLens uses a six-phase pipeline that includes code embedding, retrieval augmented CWE generation, few-shot prompt creation, multiple LLM runs, self-consistency voting and severity weighting.
Evaluation of Python, JavaScript, Java and C showed that there were a total of 15 bugs and languages that are compiled had higher risk scores because of memory safety problems. Findings suggest that LLMs can provide some benefits in semantics, but they are also subject to errors such as hallucinations and lack of consistency. This research shows that LLMs can perform better if used alongside established security data sources.
The literature review focuses on the evaluation of LLMs for tasks including test case generation, bug discovery, vulnerability discovery, test oracle generation and automated software fixing. The online survey collects the opinions of practitioners on the use, reliability and readiness of using LLMs for testing purposes. BugLens uses a six-phase pipeline that includes code embedding, retrieval augmented CWE generation, few-shot prompt creation, multiple LLM runs, self-consistency voting and severity weighting.
Evaluation of Python, JavaScript, Java and C showed that there were a total of 15 bugs and languages that are compiled had higher risk scores because of memory safety problems. Findings suggest that LLMs can provide some benefits in semantics, but they are also subject to errors such as hallucinations and lack of consistency. This research shows that LLMs can perform better if used alongside established security data sources.