Bias assessment in software systems : a comparative analysis of methods, tools, and frameworks

Abstract

Bias in software and machine learning systems has become an important concern as algorithmic decision-making is increasingly used in areas that affect individuals and society. Although numerous methods, metrics, tools, and frameworks have been proposed to evaluate bias, there is still no universally accepted approach for measuring fairness in software systems. This thesis examines how bias is measured through a comparative review of existing approaches.

The study adopts a literature-based research design and analyzes academic publications, technical reports, and official documentation related to bias measurement. The analysis focuses on theoretical fairness frameworks, commonly used fairness metrics, open-source tools, and organizational frameworks that support responsible artificial intelligence development.

The findings show that existing approaches can be classified into three main categories: fairness metrics, technical tools, and organizational frameworks. Metrics such as demographic parity and equalized odds provide quantitative methods for evaluating disparities, while tools such as AI Fairness 360, Fairlearn, and the What-If Tool support practical implementation. Organizational frameworks complement these approaches by providing ethical guidance for responsible AI development. However, no single approach is sufficient for all contexts, and different methods often reflect different interpretations of fairness.