Interaction between software developers and large language model -based tools
Bista, Bhuwan (2026)
Kandidaatintyö
2026
School of Engineering Science, Tietotekniikka
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe20260621100773
https://urn.fi/URN:NBN:fi-fe20260621100773
Tiivistelmä
LLM-based tools such as ChatGPT and GitHub Copilot have rapidly become a standard part of software development, with millions of developers now using them daily. Despite this, relatively little is known about how developers actually interact with these systems in practice. Research mostly concentrates on productivity metrics and experiments, failing to pay attention to the lived experiences of developers. Thus, this thesis examines how professional developers work with LLM tools, calibrate their trust, whether LLM tools affect their productivity, and if there is evidence of automation bias and skill atrophy.
A mixed-methods approach was applied to collect quantitative data from an online survey sent to 25 professional software developers and qualitative data from interviews conducted with seven participants. The quantitative data were subjected to descriptive statistical analysis, whereas the interview data underwent thematic analysis in accordance with the key concepts of trust calibration, automation bias, and skill atrophy.
The results demonstrate that LLM software is predominantly employed in routine and supporting tasks such as debugging and learning and is not used for software design purposes. Developers showed calibrated levels of trust that depend on verifying the output in proportion to its risk level; their calibration was affected by the frequent occurrence of incorrect output. Productivity gains were reported but depended on context; while LLM tools helped reduce coding time, they increased verification time. Although vibe coding was practised, the majority of developers recognised it as being risky and thus did not apply it in critical circumstances. When it comes to skill atrophy, the picture proved to be ambiguous, as lower-order skills such as syntax recall suffered deterioration, whereas higher-order ones such as architecture became stronger. However, the most serious issue was a possible adverse effect on juniors' skills.
The primary contribution of this thesis lies in the demonstration that professional developers actively manage the risks associated with LLM software rather than suffer from them. Additionally, the concept of trust calibration can be extended from output verification to verification of input information. The implication of these results is that LLM tools provide value to skilled professionals and pose a significant risk for beginners, highlighting the need to develop solid foundational skills among the next generation of developers.
A mixed-methods approach was applied to collect quantitative data from an online survey sent to 25 professional software developers and qualitative data from interviews conducted with seven participants. The quantitative data were subjected to descriptive statistical analysis, whereas the interview data underwent thematic analysis in accordance with the key concepts of trust calibration, automation bias, and skill atrophy.
The results demonstrate that LLM software is predominantly employed in routine and supporting tasks such as debugging and learning and is not used for software design purposes. Developers showed calibrated levels of trust that depend on verifying the output in proportion to its risk level; their calibration was affected by the frequent occurrence of incorrect output. Productivity gains were reported but depended on context; while LLM tools helped reduce coding time, they increased verification time. Although vibe coding was practised, the majority of developers recognised it as being risky and thus did not apply it in critical circumstances. When it comes to skill atrophy, the picture proved to be ambiguous, as lower-order skills such as syntax recall suffered deterioration, whereas higher-order ones such as architecture became stronger. However, the most serious issue was a possible adverse effect on juniors' skills.
The primary contribution of this thesis lies in the demonstration that professional developers actively manage the risks associated with LLM software rather than suffer from them. Additionally, the concept of trust calibration can be extended from output verification to verification of input information. The implication of these results is that LLM tools provide value to skilled professionals and pose a significant risk for beginners, highlighting the need to develop solid foundational skills among the next generation of developers.