AI or LLM assisted software testing : a mapping study

Abstract

Modern software systems are increasingly distributed, frequently updated and deployed in security-critical contexts, which reduces the effectiveness of testing approaches that rely heavily on manual effort and fragile automation. AI, particularly large language models (LLMs), is being applied to automate and support software testing activities. This study presents a systematic mapping analysis based on a Systematic Literature Review (SLR) of peer-reviewed research on AI/LLM-assisted software testing, focusing on integration strategies, adoption motivations, limitations, quality impacts and ethical considerations. Searches in IEEE Xplore, ACM Digital Library, ScienceDirect, Wiley Online Library and Google Scholar retrieved 33,572 records. After multi-stage screening and quality assessment, 34 primary studies were included and synthesized using research question driven thematic analysis. The results highlight that the current integration is mostly focused on generation. The most modern applications are those that help with unit test generation and enhancement, test suite augmentation and debugging. Other uses include finding bugs, helping with test-oracle and automating CI/CD but coverage of non-functional and specialized testing is still limited. Adoption is motivated by the need to improve coverage, reduce manual effort and accelerate feedback cycles. Key challenges include hallucinations and semantic errors in generated artifacts, weak or inconsistent oracle strategies, non-determinism and reproducibility issues and security and privacy risks associated with prompts, logs and generated code. Overall, the evidence suggests that AI/LLM assisted testing can improve software quality when supported by validation mechanisms, secure integration practices and appropriate human oversight.