Hyppää sisältöön
    • Suomeksi
    • På svenska
    • In English
  • Suomeksi
  • In English
  • Kirjaudu
Näytä aineisto 
  •   Etusivu
  • LUTPub
  • Diplomityöt ja Pro gradu -tutkielmat
  • Näytä aineisto
  •   Etusivu
  • LUTPub
  • Diplomityöt ja Pro gradu -tutkielmat
  • Näytä aineisto
JavaScript is disabled for your browser. Some features of this site may not work without it.

Calibrating linguistic risk : a Bayesian decision-analytic framework for enterprise machine translation quality estimation

Barannikova, Ekaterina (2026)

Katso/Avaa
Mastersthesis_Barannikova_Ekaterina.pdf (2.964Mb)
Lataukset: 


Pro gradu -tutkielma

Barannikova, Ekaterina
2026

School of Business and Management, Kauppatieteet

Kaikki oikeudet pidätetään.
Näytä kaikki kuvailutiedot
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi-fe20260820118551

Tiivistelmä

Machine Translation Quality Estimation (MTQE) evaluates the reliability of machine-generated translations without requiring a human reference text. In enterprise localization, such systems often provide raw and overconfident scores that fail to account for the natural uncertainty of human evaluation, a problem empirically evidenced in this study by a 12% severe inter-annotator conflict rate and moderate baseline agreement (κ=0.4051). This research proposes a Bayesian framework to quantify and manage the linguistic risks of segments being too dangerous to publish based on MTQE decision within enterprise localization.

Because annotator disagreement serves as a crucial indicator of underlying linguistic risk, reconciling human judgments is essential for accurate risk modelling. To align dual human annotations, such as binary and continuous penalties within an industrial dataset of 7,635 translation segments across 16 language pairs, a hierarchical Joint Bernoulli-Zero-Inflated Beta (ZIB) Bayesian model, constrained by a 1PL Rasch-equivalent model, was developed. This technique successfully isolated true segment-level linguistic risk, attributing nearly 73% of posterior variance to the latent linguistic signal while filtering systematic organizational and domain biases.

Benchmarking six external MTQE systems against this inferred continuous probability of linguistic failure revealed that raw MTQE scores correlate weakly with actual operational risk (maximum ρ= -0.18). To translate the raw, uncalibrated MTQE scores into reliable risk probabilities, Fractional Logistic Regression was utilized to perform soft-label calibration. While this significantly improved probabilistic reliability, reducing maximum Brier scores by 74% (from 0.1358 to 0.0348), the explained variance remained low (Efron’s pseudo-R^{2}\leq 0.067). Consequently, calibrated predictions compress into a narrow probability band (0.39 to 0.55), demonstrating that while current MTQE systems can reliably estimate aggregate cohort risk, they lack the item-level discrimination resolution required to autonomously isolate extreme translation failures.
Kokoelmat
  • Diplomityöt ja Pro gradu -tutkielmat [15783]
LUT-yliopisto
PL 20
53851 Lappeenranta
Ota yhteyttä | Tietosuoja | Saavutettavuusseloste
 

 

Tämä kokoelma

JulkaisuajatTekijätNimekkeetKoulutusohjelmaAvainsanatSyöttöajatYhteisöt ja kokoelmat

Omat tiedot

Kirjaudu sisäänRekisteröidy
LUT-yliopisto
PL 20
53851 Lappeenranta
Ota yhteyttä | Tietosuoja | Saavutettavuusseloste