Calibrating linguistic risk : a Bayesian decision-analytic framework for enterprise machine translation quality estimation
Barannikova, Ekaterina (2026)
Pro gradu -tutkielma
Barannikova, Ekaterina
2026
School of Business and Management, Kauppatieteet
Kaikki oikeudet pidätetään.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe20260820118551
https://urn.fi/URN:NBN:fi-fe20260820118551
Tiivistelmä
Machine Translation Quality Estimation (MTQE) evaluates the reliability of machine-generated translations without requiring a human reference text. In enterprise localization, such systems often provide raw and overconfident scores that fail to account for the natural uncertainty of human evaluation, a problem empirically evidenced in this study by a 12% severe inter-annotator conflict rate and moderate baseline agreement (κ=0.4051). This research proposes a Bayesian framework to quantify and manage the linguistic risks of segments being too dangerous to publish based on MTQE decision within enterprise localization.
Because annotator disagreement serves as a crucial indicator of underlying linguistic risk, reconciling human judgments is essential for accurate risk modelling. To align dual human annotations, such as binary and continuous penalties within an industrial dataset of 7,635 translation segments across 16 language pairs, a hierarchical Joint Bernoulli-Zero-Inflated Beta (ZIB) Bayesian model, constrained by a 1PL Rasch-equivalent model, was developed. This technique successfully isolated true segment-level linguistic risk, attributing nearly 73% of posterior variance to the latent linguistic signal while filtering systematic organizational and domain biases.
Benchmarking six external MTQE systems against this inferred continuous probability of linguistic failure revealed that raw MTQE scores correlate weakly with actual operational risk (maximum ρ= -0.18). To translate the raw, uncalibrated MTQE scores into reliable risk probabilities, Fractional Logistic Regression was utilized to perform soft-label calibration. While this significantly improved probabilistic reliability, reducing maximum Brier scores by 74% (from 0.1358 to 0.0348), the explained variance remained low (Efron’s pseudo-R^{2}\leq 0.067). Consequently, calibrated predictions compress into a narrow probability band (0.39 to 0.55), demonstrating that while current MTQE systems can reliably estimate aggregate cohort risk, they lack the item-level discrimination resolution required to autonomously isolate extreme translation failures.
Because annotator disagreement serves as a crucial indicator of underlying linguistic risk, reconciling human judgments is essential for accurate risk modelling. To align dual human annotations, such as binary and continuous penalties within an industrial dataset of 7,635 translation segments across 16 language pairs, a hierarchical Joint Bernoulli-Zero-Inflated Beta (ZIB) Bayesian model, constrained by a 1PL Rasch-equivalent model, was developed. This technique successfully isolated true segment-level linguistic risk, attributing nearly 73% of posterior variance to the latent linguistic signal while filtering systematic organizational and domain biases.
Benchmarking six external MTQE systems against this inferred continuous probability of linguistic failure revealed that raw MTQE scores correlate weakly with actual operational risk (maximum ρ= -0.18). To translate the raw, uncalibrated MTQE scores into reliable risk probabilities, Fractional Logistic Regression was utilized to perform soft-label calibration. While this significantly improved probabilistic reliability, reducing maximum Brier scores by 74% (from 0.1358 to 0.0348), the explained variance remained low (Efron’s pseudo-R^{2}\leq 0.067). Consequently, calibrated predictions compress into a narrow probability band (0.39 to 0.55), demonstrating that while current MTQE systems can reliably estimate aggregate cohort risk, they lack the item-level discrimination resolution required to autonomously isolate extreme translation failures.
