An empirical evaluation of large language models for Python code generation in software quality assurance

Abstract

Large language models are increasingly used as programming assistants, but their generated code still needs to be examined from the perspective of software quality assurance. This thesis evaluates Python code generated by two selected models, GPT-5.5 and Qwen3:8B, on a controlled set of twenty short programming tasks. The tasks cover input validation, parsing, data structures, algorithmic logic, exception handling, and cache simulation.

The study uses an automated Python evaluation script to load each generated solution, execute predefined functional tests, record pass and fail outcomes, and calculate code metrics using cyclomatic complexity and maintainability index. The experiment produced summary and detailed result files that were used as the empirical basis for the analysis.

The results show a clear difference between the two evaluated models. GPT-5.5 passed all twenty tasks and all 108 executed test cases. Qwen3:8B fully passed 17 out of 20 tasks and passed 96 out of 102 executed test cases, with additional failure caused by a syntax error in one task file. The findings suggest that LLM-generated code can be useful for short programming tasks, but it should still be treated as untrusted code until it has passed automated tests and human review.