Who can we trust? LLM-as-a-jury for Comparative Assessment 事件

Who can we trust? LLM-as-a-jury for Comparative Assessment arXiv:2602.16610v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment