Migrating from AWS S3

1. Inventory — before anything moves

The endpoint swap is trivial. What is not trivial is discovering, three weeks in, that something in your stack depended on a feature that is not here. Go through this list first; it is short, and every item maps to something specific.

2. Provision

1. Create the buckets. One per S3 bucket you are moving is the simplest mapping; pick hot or cool per bucket, not per object.
2. Create one credential per service that will talk to storage, scoped to the buckets that service needs. Do not reuse a single key everywhere — you are already re-doing access control, so do it properly while you are here.
3. Create one temporary, wide-scoped credential for the migration itself, with an expiry a few days out. It should stop working on its own when you are done.
4. If anything uploads from a browser, set the bucket’s CORS rules now, before you test — a missing rule looks exactly like a broken signature.

3. Configure

The endpoint, region, credentials and bucket map directly from your AWS configuration.

export AWS_ACCESS_KEY_ID=PCAK00EXAMPLEKEYID00
export AWS_SECRET_ACCESS_KEY=<your secret access key>
export AWS_ENDPOINT_URL=https://s3.canada.popcloud.ca
export AWS_REGION=canada
export POPCLOUD_BUCKET=pc-your-org-media

Choose path-style for dotted bucket names or clients without nested wildcard TLS; virtual-hosted addressing is supported on the CA production endpoint. Find your stack on the integrations page for the exact setting names.

4. Copy the data

Copy provider to provider, not through your laptop. rclone streams between the two and verifies by hash, which matters more than speed.

# Streams provider to provider — nothing lands on the local disk. Define both
# remotes in rclone.conf first ([s3-old] for the source, [popcloud] for us).
rclone copy s3-old:legacy-bucket "popcloud:${BUCKET}" \
  --checksum \
  --transfers 16 \
  --checkers 32 \
  --s3-upload-concurrency 8 \
  --s3-chunk-size 32M \
  --stats 30s \
  --log-file migration.log

Run it while your application is still writing to S3. This first pass does not need to be consistent — it needs to move the bulk. Size the concurrency to your egress budget rather than your patience; see copying your data for the details, including resumption and the two key shapes that will trip a bulk copy.

5. Cut over

Three strategies. Pick by how much downtime you can spend, not by elegance.

Big bang

Stop writes, run a final incremental sync, flip the configuration, start again.

# Cutover pass: only what changed since the bulk copy. Run it with writes to the
# source stopped, and it finishes in minutes rather than hours.
rclone sync s3-old:legacy-bucket "popcloud:${BUCKET}" \
  --checksum \
  --transfers 16 \
  --stats 30s

• Downtime: minutes to an hour, depending on the delta.
• Best for: most applications. It is the simplest thing that works.
• Rollback: flip the configuration back. The old data is untouched.

Dual-write

Write to both for a period, read from S3, then switch reads. Removes the downtime and gives you a live comparison.

• Cost: an application change, and both bills, for the overlap.
• Best for: systems where a write window cannot be arranged, or where you want to watch error rates on real traffic before committing.
• Rollback: switch reads back. Both stores are current.

Read-through

Switch writes immediately; on a read miss, fetch from S3, store, and serve. The old bucket drains as traffic hits it.

• Cost: read-path complexity, and a long tail that never migrates itself.
• Best for: very large buckets where most objects are never read again.
• Finish it anyway. Run the bulk copy in the background or you will be paying for two stores indefinitely.

6. Verify

Prove it, do not assume it. Three checks, in increasing order of confidence:

1. Count and size. Object counts and total bytes match per prefix.
2. Hashes. Compare both sides by checksum. This is the one that matters.
3. Application-level. Fetch a sample through your own code paths — signed URLs, CDN, image resizing — not just through the storage API.

# Re-run after the copy. --one-way ignores anything already on the destination,
# so it answers exactly one question: did everything on the source arrive?
rclone check s3-old:legacy-bucket "popcloud:${BUCKET}" --checksum --one-way

7. Decommission — and how to go back

Keep the S3 bucket for at least one full backup cycle after cutover. Delete the migration credential, or let it expire. Then re-check the AWS bill a month later: egress and request charges usually keep arriving for a while after the storage line drops.

Rolling back is deliberately boring: point your configuration back at S3. As long as you have not deleted the source bucket, nothing else is required — which is exactly why the source bucket stays for a cycle.