Prompt Optimization Is a Coin Flip: Diagnosing When It Helps in Compound AI Systems 事件

Prompt Optimization Is a Coin Flip: Diagnosing When It Helps in Compound AI Systems arXiv:2604.14585v2 Announce Type: replace-cross Abstract: Prompt optimization in compound AI systems is statistically indistinguishable from a coin flip: across 72 optimization runs on Claude Haiku 4.5 (6 methods $\times$ 4 tasks $\times$ 3 repeats), 49% score below zero-shot; on Amazon Nova Lite, the failure rate is even higher. Yet on one task, all six methods improve over zero-shot by up to $+6.8$ points. Wha