Seamless Distributed Training on Amazon EKS with NVRx
AWS demonstrates the integration of the NVIDIA NVRx extension with Amazon EKS and PyTorch FSDP to prevent interruptions in large-scale AI training and increase GPU efficiency.
To prevent hardware and network issues during large-scale distributed artificial intelligence training, AWS has announced how to integrate the NVIDIA Resiliency Extension (NVRx) technology with Amazon EKS and PyTorch FSDP.
Interruptions in Distributed Training
In large-scale distributed training jobs that last for days, disruptions such as network outages, memory errors, and hardware failures inevitably arise.
A single GPU failure causes time-outs even in healthy workers, leading to wasted expensive GPU hours.
NVRx Integration and Key Features
To overcome these challenges, AWS combined the NVIDIA Resiliency Extension (NVRx) system with PyTorch FSDP in the Amazon Elastic Kubernetes Service environment.
NVRx provides advanced components that deliver application-level resilience without the need for custom kernels or PyTorch modifications.
Asynchronous Checkpoints
The asynchronous checkpoint feature eliminates bottlenecks caused by simultaneous saving operations by offloading the state dictionary to a background process.
Each rank writes its own shard directly, minimizing the waiting times and time losses seen in traditional methods.
In-Process and In-Job Restart
In-process restart allows health checks to be performed without terminating the Python process during transient errors, enabling the system to resume from the latest checkpoint.
In-job restart cleans up survivors during out-of-memory or critical OS-level crashes, allowing new workers to be respawned within the same job.
Amazon EKS Infrastructure and Performance
The EKS environment provides a robust infrastructure for NCCL operations with high network bandwidth p5.48xlarge instances.
The Amazon FSx for Lustre file system is mounted inside each pod, providing shared POSIX support for distributed checkpoint read and write operations.