Copying your data

Moving the bytes, resuming when it breaks, and proving afterwards that nothing was lost.

Pick the tool

Tool	Best at	Not for
rclone	Provider to provider without a local round trip; hash verification; resuming. The default choice for a migration.	Nothing much — it is slower than s5cmd on very large object counts.
aws s3 sync	Local directories up; small one-off copies; already installed everywhere.	Cross-provider copies — it needs two profiles and still pulls the bytes through your machine.
s5cmd	Millions of small objects, where per-request overhead dominates.	Not documented here yet — the configuration table on the integrations page is enough to set it up.

Provider to provider

Define both remotes, then stream between them. Nothing lands on local disk, so the transfer is bounded by the two providers rather than by your machine.

Configure

verified

# ~/.config/rclone/rclone.conf
cat >> "$RCLONE_CONFIG" <<CONF
[popcloud]
type = s3
provider = Other
endpoint = ${AWS_ENDPOINT_URL}
region = ${AWS_REGION}
access_key_id = ${AWS_ACCESS_KEY_ID}
secret_access_key = ${AWS_SECRET_ACCESS_KEY}
# Recommended for predictable custom-endpoint behavior. PopCloud also accepts
# virtual-hosted requests for ordinary single-label bucket names.
force_path_style = true
# rclone otherwise tries to create a missing destination bucket, which PopCloud
# answers with 405 — buckets are created in the dashboard.
no_check_bucket = true
CONF

Copy from another provider

not run in CI

# Streams provider to provider — nothing lands on the local disk. Define both
# remotes in rclone.conf first ([s3-old] for the source, [popcloud] for us).
rclone copy s3-old:legacy-bucket "popcloud:${BUCKET}" \
  --checksum \
  --transfers 16 \
  --checkers 32 \
  --s3-upload-concurrency 8 \
  --s3-chunk-size 32M \
  --stats 30s \
  --log-file migration.log

From a local directory

shell

verified

aws s3 cp ./local-file.bin "s3://$BUCKET/${PREFIX}key.bin" --profile popcloud

# A whole directory, in parallel, skipping what is already there.
aws s3 sync ./public "s3://$BUCKET/${PREFIX}public/" --profile popcloud

Sizing the transfer

Three numbers matter, and only one of them is speed. Start conservative, watch the first ten minutes, then raise concurrency — a copy that saturates your source provider’s rate limits finishes later than one that does not.

Setting	Start at	Why
--transfers	16	Objects in flight. Raise for many small objects, lower if you see throttling.
--checkers	32	Comparisons in flight. Cheap; the listing side is usually not the bottleneck.
--s3-chunk-size	32M	Multipart part size. Larger means fewer requests and more memory per transfer; 8–64 MiB is the useful range.

Budget for egress at the source. Moving out of AWS costs per gigabyte, and that one-off charge is usually the largest single line of a migration.

When it breaks — and it will

Resume by re-running the same command. Both tools compare before transferring, so a repeat pass moves only what is missing. Always log to a file: without --log-file a multi-hour run tells you nothing about what it skipped.
Throttling shows up as slow transfers with retries rather than hard failures. Lower --transfers rather than restarting.
Individual key failures do not stop the run. That is the dangerous case — the summary says success and the count is short.

Blocker

Keys containing . or .. path segments are rejected

You’ll see: 403 SignatureDoesNotMatch for a key with a dot segment in it, such as `./relative/../path.bin`, while every other key in the same run works.
Why: The URL is normalised somewhere between signing and verification, so the path that gets verified is not the path that was signed. This is a defect.
Instead: Normalise keys before uploading — collapse `.` and `..` segments yourself. This matters most when keys are derived from filesystem paths during a migration; a bulk copy will otherwise fail on a small, unpredictable subset of objects.

Worth knowing

Only the STANDARD storage class exists

You’ll see: 501 NotImplemented when a request sets x-amz-storage-class to GLACIER, STANDARD_IA, INTELLIGENT_TIERING or similar.
Why: Storage economics are expressed as a bucket's tier (hot or cool), chosen when the bucket is created, rather than per object.
Instead: Leave the storage class unset. If your tool sets one by default, point it at STANDARD. To store colder data, create a cool-tier bucket and write there. RestoreObject does not exist either, because nothing is ever archived.

Restore archived objects before copying, not after

Anything in Glacier or a deep-archive class reads as a stub. Copy it and you have faithfully migrated a stub. Restore first, confirm, then copy.

Proving it worked

Hash comparison, one-way. Ignores anything already on the destination and answers one question: did everything on the source arrive?
Counts per prefix. Catches a whole prefix that was never listed — a permission problem at the source, not a transfer problem.
A sample through your application. Signed URLs, CDN, thumbnails. Storage-level equality does not prove your app can serve the object.

shell

not run in CI

# Re-run after the copy. --one-way ignores anything already on the destination,
# so it answers exactly one question: did everything on the source arrive?
rclone check s3-old:legacy-bucket "popcloud:${BUCKET}" --checksum --one-way

Then keep the source for at least one backup cycle. It is the entire rollback plan, and it costs one month of storage.