# AWS and Qumulo show cross-Region training without copying data

> AWS's ML blog detailed a validated setup pairing SageMaker HyperPod with Qumulo storage, letting a cluster in one Region train on data in another at matching throughput.

- **Topic**: Models
- **Published**: 2026-09-27T00:55:51.722Z
- **Canonical URL**: https://highsignal.sh/stories/aws-and-qumulo-show-cross-region-training-without-copying-data-703fb5f7

## Why It Matters

Data gravity often forces teams to either replicate petabytes across Regions or absorb latency on every read. AWS says the pairing removes that trade-off without code changes.

## Key Findings & Analysis

### What AWS validated

> - AWS described pairing **Amazon SageMaker HyperPod** with **Qumulo's Cloud Native Qumulo (CNQ) and Cloud Data Fabric (CDF)**, so training jobs read datasets stored in another AWS Region or on-premises without copying data or changing code (AWS Machine Learning Blog).
> - In the test, the hub cluster ran in US East (Ohio) with the data; the spoke cluster ran in US West (Oregon) and read remotely through CDF at 60 ms network latency (AWS).
> - After a warmup, the spoke matched the hub's throughput: 115–117 samples/sec, converging within the first 100–150 batches (AWS). AWS says that warmup is under 1 percent of wall-clock time at 10,000+ batches and under 0.1 percent beyond 100,000 (AWS).
> - AWS reported 80–90 percent GPU utilization during that initial phase, converging to 98–100 percent, and full 99 percent GPU utilization in single-Region training with sub-3 ms data operations (AWS).

All performance figures are AWS's own validation results, not independent benchmarks.

### How it works

> - Compute nodes mount a local Qumulo instance over NFS in each Region; the spoke retrieves data from the hub over VPC peering, so nodes don't read cross-Region directly (AWS).
> - Qumulo's CDF uses predictive caching (NeuralCache) to learn data-loader access patterns and serve reads from local NVMe, which AWS says makes cross-Region latency transparent after warmup (AWS).
> - Jobs are submitted in the spoke Region through an Amazon EKS orchestrator; HyperPod supplies managed infrastructure with health checks, node replacement, and checkpoint recovery (AWS).

Verification and operational details beyond AWS's published results aren't available in the supplied evidence.

## Primary Sources & Citations

- [Multi-Region training with Amazon SageMaker HyperPod and Qumulo](https://aws.amazon.com/blogs/machine-learning/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo/) — *AWS Machine Learning Blog* (Primary source)

---

[← Back to Headlines](https://highsignal.sh/) | [Daily Brief](https://highsignal.sh/brief) | [All stories](https://highsignal.sh/latest)