Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory 事件

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory arXiv:2602.00521v2 Announce Type: replace Abstract: While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge