Uncertainty quantification via Iterative conformal estimation of mixture-of-agents

Abstract

Although large language models are widely used for answering complex questions and generating factual text, they often produce inaccurate, fabricated, or uncertain claims without signalling uncertainty. This poses a major challenge for organisations seeking to deploy such systems in high-stakes domains such as healthcare, where mistakes can have serious consequences. This thesis proposes a post-hoc framework, Conformal Mixture-of-Agents (C-MoA), which provides formal statistical reliability guarantees for the outputs of multi-agent language model systems without requiring retraining or access to model parameters. The framework decomposes responses into atomic claims, scores them based on agreement among language model agents, filters low-confidence claims, and applies a calibrated threshold to control the error rate at a user-specified level. The framework is evaluated on two public benchmark datasets and an internal healthcare dataset developed at Sogeti. For long-form biography generation, the precision of retained claims increases from 0.41 to 0.83. Moreover, perfect precision (1.00) is achieved on the internal healthcare dataset when an LLM-based judge is used for verification. Cross-domain experiments show that calibration can be transferred across domains without recalibration. The results confirm the importance of atomic claim decomposition and multi-agent agreement, with the removal of either reducing precision by more than 35 percentage points. Overall, conformal prediction provides a practical reliability layer for multi-agent language generation systems without modifying the underlying models.