A new study on arXiv challenges a popular idea in legal AI: that fusing multiple uncertainty tools (evidence graphs, Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline can improve case-outcome predictions. Tested on 1,000 real cases from the European Court of Human Rights (ECHR), the researchers find that a frontier LLM used directly is the strongest single discriminator – the fusion pipeline yields no improvement on discrimination (AUROC around 0.83).
Worse, naively composing an LLM with Bayesian odds and Dempster-Shafer more than doubles calibration error (ECE from about 0.16 to 0.46) via a prior-mismatch mechanism that replicates across both tested models (Claude Opus 4.8 and GPT-5.5). Dempster-Shafer fusion is even actively unsafe on long chains, committing confidently to wrong labels at below-chance accuracy – the researchers recommend removing it.
The real value lies elsewhere: in a conformal selective-prediction layer that decides which cases the system automates and which it escalates to humans. After removing Dempster-Shafer, recalibrating, and applying class-conditional risk control, the tuned engine achieves 96.8 percent accuracy on auto-cleared cases, with only 0.5 percent errors escaping and 96.3 percent of errors caught for review – versus 85.9 / 3.8 / 72.1 for an untuned baseline.
For decision-makers in legal technology and automation, the point is clear: complex uncertainty fusion is not a miracle tool for sharper predictions. But used correctly – as a filter that routes difficult cases to humans – it can enable reliable automation with low error risk. This is a matter of calibrated trust, not sharper prediction.
Conclusion: Don't invest in complex fusion pipelines to improve prediction – use them for selective automation, where they genuinely lift accuracy and reduce error risk.
