2.3 Hybrid Data Transfer with Snow Family & AWS DataSync

2.3 Hybrid Data Transfer with DataSync and Snow Family

Moving large datasets into AWS is a scheduling and risk problem, not just a service-name question. A data engineer must measure the source, usable network throughput, daily change rate, metadata requirements, outage window, and destination before choosing an online or physical workflow.

Current Snowball availability

AWS states that Snowball Edge is no longer available to new customers and that no Snow Family device can be newly ordered by those customers. Existing Snowball Edge customers can continue using the service. New customers should evaluate AWS DataSync for online movement, AWS Data Transfer Terminal for secure high-throughput physical locations, or AWS Partner solutions. The DEA-C01 in-scope list still names Snow Family, so understand its physical-transfer pattern, but do not recommend a new device order without establishing that the account is eligible.

Start with a transfer-time calculation

Use this baseline:

transfer seconds = data bytes × 8 / usable bits per second

Usable throughput is lower than the link rate. Subtract normal utilization and allow for protocol overhead, small-file overhead, retransmission, and source read limits. A 600 TB source over only 30 Mbps of usable capacity takes roughly five years before overhead. A 10 Gbps path would reduce the theoretical time dramatically, but the storage array, firewall, agent, and destination must all sustain that rate.

Also calculate the change rate. If a 100 TB migration takes ten days and the source changes by 2 TB daily, the cutover plan must synchronize the 20 TB delta. A common sequence is initial copy, repeated incremental tasks, application freeze, final delta task, validation, then DNS or connection cutover.

AWS DataSync for online transfer

DataSync is a managed data movement service. An on-premises transfer commonly uses a DataSync agent to read NFS, SMB, Hadoop Distributed File System, or object storage and write to a supported destination such as Amazon S3, Amazon EFS, or Amazon FSx. Service-to-service transfers can avoid an on-premises agent when the documented location pair supports it.

Important task choices include:

• Schedule and bandwidth: Run one-time or recurring tasks and limit bandwidth when the migration shares a production link.
• Transfer mode: Copy changed data rather than recopying an entire source on every execution.
• Metadata: Select preservation options appropriate to the source and target. POSIX ownership and permissions matter for NFS-to-EFS workloads; object metadata semantics differ.
• Verification: Choose the verification level that meets the integrity requirement. Verification consumes time and I/O, so include it in the window.
• Filtering: Include or exclude paths so a task moves only the intended dataset.
• Monitoring: Send task metrics and logs to CloudWatch and alarm on errors or a growing transfer backlog.

DataSync traffic can use the internet, a virtual private network, or AWS Direct Connect. Private routing changes the network path but does not remove the need for service endpoints, security groups, routing, and DNS resolution.

Physical transfer pattern for eligible Snowball customers

For an existing eligible customer, Snowball Edge can provide local storage and optional edge compute while the device is disconnected from AWS. The workflow is: create an import job, receive and unlock the device with the manifest and unlock code, copy data locally, validate the job, return the device, and allow AWS to import the data into S3. Data on the device is encrypted, and the device includes tamper-resistant controls.

Physical shipment is not continuous replication. Files that change after the copy require another shipment or, more commonly, a DataSync final delta. Never compare only network-transfer time with copy time; include order lead time, local copy, shipping, AWS ingestion, validation, and any export-control or Region restrictions.

Decision table

Exam method

First classify the requirement as migration, continuous replication, or hybrid access. Then identify protocol and metadata needs. Finally calculate the window and apply the current availability rule. A distractor that orders Snowball for an unspecified new customer is incomplete even when the dataset is enormous.