An empirical study of LLM-assisted software testing : a practitioner’s perspective

Abstract

Context: Rapidly growing software systems due to practices such as Agile, DevOps and Continuous Integration / Continuous Deployment (CI/CD) pipeline have increased the pressure for software testing. The conventional testing procedures fail to cope with efficiency and adaptability issues, along with the coverage issue. Over the past years, Large Language Models (LLMs) have gained significant attention as potential aids for many different types of software testing activities, including creating test cases, debugging and documenting test cases. Objective: This research empirically investigates how LLM-assisted testing tools are currently used and integrated into industrial software testing workflows, the challenges and limitations practitioners encounter in their use, practitioners' perceptions of their impact on software quality and the factors that influence adoption alongside practices that support their responsible implementation. Methodology: A mixed-method research approach has been adopted to collect quantitative and qualitative data through online survey and semi-structured interviews conducted from software practitioners. The collected data has been analyzed to determine trends and patterns of practitioners' insight regarding the use of LLMs in software testing. Results: According to results, there is an increase in the use of LLM tools for software testing. The majority of practitioners have stated that their productivity, understanding of code and efficiency in testing have improved since they started using these tools. However, there are still numerous challenges associated with these tools, including concerns regarding their accuracy and reliability, the lack of contextual understanding and the difficulties associated with integrating LLM into existing processes. Results show that LLMs are primarily being used as complementary tools and that human validation is mandatory for acceptance of LLM output. Conclusion: There are significant benefits of LLM-assisted software testing for improving software development efficiency and quality. Successful adoption of LLMs will require hybrid workflows (human-AI), an organization’s readiness to accept change and clear implementation strategies. Without overcoming current limitations, LLMs will not be used for reliable and effective results in real-world software testing situations.