Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why 事件

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why arXiv:2606.00093v1 Announce Type: new Abstract: Validating an LLM judge against human annotations usually means reporting several agreement statistics: accuracy, precision, recall, $F_1$, Cohen's $\kappa$, and one or more rank correlations. A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling, and those choices rarely stated. Fo