NVIDIA Releases Open-Source Recovery Service for AI
NVIDIA Recovery detects interruptions in distributed AI training jobs and coordinates automated recovery across clusters.
Updated on Nov. 3, 2026 in Data Centers
NVIDIA introduced NVIDIA Recovery on Monday, November 2, as an open-source service in its DSX OS portfolio for detecting interruptions in distributed AI training. The experimental service combines application logs, GPU metrics, and scheduler signals to identify crashes, hangs, and stragglers, and is licensed under Apache 2.0.
Why it matters
Interruptions in large-scale AI training runs can waste significant compute time, and automated detection and recovery aims to reduce that loss. Because the service is experimental and not ready for production, its real-world effect on training efficiency remains to be tested.
On one monitored cluster, mean anomaly detection latency was 69 seconds and the average time to an automated recovery action was 94 seconds across about 8,550 detections from July 9 through October 9. The measurements come from a single cluster, and the source does not state how they would translate to other environments.
The players
NVIDIA
NVIDIA is a semiconductor and AI computing company whose GPUs are widely used to train large-scale artificial intelligence models.
The details
NVIDIA Recovery identifies interruptions by combining application logs, GPU metrics, and scheduler signals, then coordinates a recovery response across the cluster. Documented support covers Slurm and Slurm on Kubernetes through Slinky, while native Kubernetes workload recovery has not been implemented.
Timeline
July 9 through October 9: Cluster monitoring period recording anomaly detections.
Monday, November 2: NVIDIA Recovery introduced as an open-source service.
The Tech Race
NVIDIA Recovery builds on the established Slurm scheduler and its Kubernetes integration through Slinky, extending them with automated interruption handling for GPU training jobs. The move positions NVIDIA's DSX OS portfolio alongside other infrastructure tools competing to keep large AI training clusters running efficiently.
For developers running distributed AI training, NVIDIA Recovery offers an open-source option to detect crashes, hangs, and stragglers without building that logic from scratch. Because the service is experimental and not ready for production, teams should evaluate it in test environments before relying on it for production training jobs.
The takeaway
Automated interruption detection can cut the time lost when AI training jobs fail, but teams should verify performance on their own infrastructure before adopting experimental tools.
Further reading
For more on the infrastructure that runs these training clusters, see Data Centers.
