What AWS validated
> - AWS described pairing **Amazon SageMaker HyperPod** with **Qumulo's Cloud Native Qumulo (CNQ) and Cloud Data Fabric (CDF)**, so training jobs read datasets stored in another AWS Region or on-premises without copying data or changing code (AWS Machine Learning Blog). > - In the test, the hub cluster ran in US East (Ohio) with the data; the spoke cluster ran in US West (Oregon) and read remotely through CDF at 60 ms network latency (AWS). > - After a warmup, the spoke matched the hub's throughput: 115–117 samples/sec, converging within the first 100–150 batches (AWS). AWS says that warmup is under 1 percent of wall-clock time at 10,000+ batches and under 0.1 percent beyond 100,000 (AWS). > - AWS reported 80–90 percent GPU utilization during that initial phase, converging to 98–100 percent, and full 99 percent GPU utilization in single-Region training with sub-3 ms data operations (AWS).
All performance figures are AWS's own validation results, not independent benchmarks.
How it works
> - Compute nodes mount a local Qumulo instance over NFS in each Region; the spoke retrieves data from the hub over VPC peering, so nodes don't read cross-Region directly (AWS). > - Qumulo's CDF uses predictive caching (NeuralCache) to learn data-loader access patterns and serve reads from local NVMe, which AWS says makes cross-Region latency transparent after warmup (AWS). > - Jobs are submitted in the spoke Region through an Amazon EKS orchestrator; HyperPod supplies managed infrastructure with health checks, node replacement, and checkpoint recovery (AWS).
Verification and operational details beyond AWS's published results aren't available in the supplied evidence.
Sources
- Multi-Region training with Amazon SageMaker HyperPod and Qumulo
AWS Machine Learning Blog · Primary source ·