Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents 事件

Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents arXiv:2606.10315v1 Announce Type: new Abstract: LLM-as-judge is the default instrument for evaluating conversational agents, yet its reliability is almost always reported as agreement with human ratings, not recall of real defects. We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems its built-in LLM judge catches, using exhaustive human transcript