Sentiment analysis of episodic volunteers in open source software projects
Pham Tran Hien, Dung (2026)
Diplomityö
2026
School of Engineering Science, Tietotekniikka
Kaikki oikeudet pidätetään.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe2026050639752
https://urn.fi/URN:NBN:fi-fe2026050639752
Tiivistelmä
Open source software (OSS) projects rely heavily on volunteer contributors, among whom episodic volunteers represent a significant yet understudied segment of the contributor population. While prior research has examined contributor retention and participation patterns in OSS communities, little attention has been given to understanding the behavioural and communicative dynamics of episodic volunteers, particularly as expressed through the sentiment of their pull request (PR) interactions.
This study investigates episodic volunteer behaviour across 20 open source GitHub projects by analysing the sentiment in PR communication, including PR descriptions, issue comments, review comments, and formal reviews. A pre-trained RoBERTa model fine-tuned on the GoEmotions dataset is applied to assign one dominant emotion label from 28 categories to each English-language comment, after filtering out bot-generated and non-English content. Model reliability was validated against a stratified sample of 280 manually annotated comments, yielding an exact-match accuracy of 84.3% and a weighted F1 score of 0.84. Episodic volunteers are operationally defined as contributors who are not members of the project's core team, are not compensated for their contributions, and do not exhibit sustained, high-volume contribution patterns over the observation period, identified through email domain classification and organisational membership analysis.
The findings show that episodic volunteers account for a mean of 83.5% of contributors across the 20 projects, yet their pull requests achieve a mean merge rate of 66.7% compared with the overall project average of 82.3%, and they account for a mean of only 13.5% of merged lines of code. Episodic volunteers communicate predominantly in a neutral register, with gratitude as the most prevalent non-neutral emotion in every project. Reviewer tone toward episodic volunteer pull requests is not harsher on rejection; across 19 of 20 projects, rejection is associated with an increase in gratitude and curiosity rather than negative emotions. No statistically significant correlation was found between project-level sentiment climate and episodic volunteer activity, indicating that aggregate sentiment metrics are too coarse to capture the interpersonal dynamics relevant to episodic contributor retention.
The findings provide actionable insights for OSS maintainers, including the importance of welcoming reviewer language on rejected contributions, responsive review queues for episodic contributor retention, and first-interaction-targeted sentiment monitoring rather than project-wide sentiment aggregates.
This study investigates episodic volunteer behaviour across 20 open source GitHub projects by analysing the sentiment in PR communication, including PR descriptions, issue comments, review comments, and formal reviews. A pre-trained RoBERTa model fine-tuned on the GoEmotions dataset is applied to assign one dominant emotion label from 28 categories to each English-language comment, after filtering out bot-generated and non-English content. Model reliability was validated against a stratified sample of 280 manually annotated comments, yielding an exact-match accuracy of 84.3% and a weighted F1 score of 0.84. Episodic volunteers are operationally defined as contributors who are not members of the project's core team, are not compensated for their contributions, and do not exhibit sustained, high-volume contribution patterns over the observation period, identified through email domain classification and organisational membership analysis.
The findings show that episodic volunteers account for a mean of 83.5% of contributors across the 20 projects, yet their pull requests achieve a mean merge rate of 66.7% compared with the overall project average of 82.3%, and they account for a mean of only 13.5% of merged lines of code. Episodic volunteers communicate predominantly in a neutral register, with gratitude as the most prevalent non-neutral emotion in every project. Reviewer tone toward episodic volunteer pull requests is not harsher on rejection; across 19 of 20 projects, rejection is associated with an increase in gratitude and curiosity rather than negative emotions. No statistically significant correlation was found between project-level sentiment climate and episodic volunteer activity, indicating that aggregate sentiment metrics are too coarse to capture the interpersonal dynamics relevant to episodic contributor retention.
The findings provide actionable insights for OSS maintainers, including the importance of welcoming reviewer language on rejected contributions, responsive review queues for episodic contributor retention, and first-interaction-targeted sentiment monitoring rather than project-wide sentiment aggregates.