From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs 事件

From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs arXiv:2605.09370v2 Announce Type: replace-cross Abstract: Large-scale AI training is now fundamentally a distributed systems problem, and hardware failures have become routine operating conditions rather than rare exceptions. Public operational evidence from production training clusters, however, remains scarce. This technical report presents an empirical analysis of a 63-node NVIDIA B200 production cluster (504