Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators 事件

Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators arXiv:2606.07874v1 Announce Type: new Abstract: LLMs-as-judges are the only way to evaluate safety at scale. Despite their importance, LLM-judges themselves are rarely evaluated beyond human agreement in simple, static benchmarks. We therefore investigate two under-explored but crucial properties of LLMs-as-judges: their susceptibility to relying on in context-information, and their steerability to differing saf