Leveraging large language models for automated end-to-end testing : an empirical analysis of automated testing tools through experiments

Abstract

This study empirically evaluates the performance of automated testing tools. Due to the limitations of manual testing, automated testing is becoming inevitable. Conducted using state-of-the-art methods, the study achieves three critical objectives: demonstrates end-to-end testing of a software application, integrates LLMs into automated testing, and empirically analyzes/compares automated testing tools. The study employs an experimental strategy to conduct end-to-end testing of a software system using selected automation testing tools. The data is collected through quantitative observation and automated data logging. The testing tools excel at end-to-end testing, with significant differences in stability, and at validating dynamic UI behaviours. The non-deterministic nature of the UI, network conditions, browser scheduling, and event loop timing may have affected the performance of the testing frameworks, especially during API data fetching and rendering. These limitations are mitigated using the best-known techniques. The crucial implications are: an automated testing tool tightly integrated with the browser offers superior reliability; a keyword-driven tool, when coupled with the right libraries, offers a viable testing option, especially for cross-team use; and a versatile cross-browser testing tool offers a viable alternative for end-to-end testing. This study provides theoretical and practical contributions to software engineering.