TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
We identify two fundamental inconsistencies in LLM-as-a-judge evaluation: Score-Comparison inconsistency and Pairwise Transitivity inconsistency. We propose TrustJudge, a probabilistic framework with distribution-sensitive scoring and likelihood-aware aggregation to alleviate these issues and improve reliability of automated assessment.