STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems 事件

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems arXiv:2605.02122v2 Announce Type: replace-cross Abstract: Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation. Majority vote discards annotator reliability and item-level ambiguity, often yielding unstable comparisons across annotator subsets. We introduce STABLEVAL, a disagreement-awa