tool

rclone

The migration workhorse — provider to provider, verified by hash

verified in CIv1.65+rclone documentation ↗

Install

shell

not run in CI

curl https://rclone.org/install.sh | sudo bash

Configure

Configure the endpoint, region, credential and bucket in this tool’s vocabulary. Path-style addressing is recommended for dotted bucket names and clients without nested wildcard TLS, but virtual-hosted addressing is also supported.

In rclone	Set it to
endpoint	https://s3.canada.popcloud.ca
region	canada
access_key_id	PCAK00EXAMPLEKEYID00
secret_access_key	<your secret access key>
force_path_style	true

Environment

example values

export AWS_ACCESS_KEY_ID=PCAK00EXAMPLEKEYID00
export AWS_SECRET_ACCESS_KEY=<your secret access key>
export AWS_ENDPOINT_URL=https://s3.canada.popcloud.ca
export AWS_REGION=canada
export POPCLOUD_BUCKET=pc-your-org-media

These are example values. Sign in and every snippet on this site fills in with your own endpoint, key and bucket.

config

verified

# ~/.config/rclone/rclone.conf
cat >> "$RCLONE_CONFIG" <<CONF
[popcloud]
type = s3
provider = Other
endpoint = ${AWS_ENDPOINT_URL}
region = ${AWS_REGION}
access_key_id = ${AWS_ACCESS_KEY_ID}
secret_access_key = ${AWS_SECRET_ACCESS_KEY}
# Recommended for predictable custom-endpoint behavior. PopCloud also accepts
# virtual-hosted requests for ordinary single-label bucket names.
force_path_style = true
# rclone otherwise tries to create a missing destination bucket, which PopCloud
# answers with 405 — buckets are created in the dashboard.
no_check_bucket = true
CONF

Use it

List a prefix

shell

verified

rclone ls "popcloud:${BUCKET}/${PREFIX}"
rclone lsd popcloud:                 # the buckets this credential can see

Delete an object

shell

verified

rclone purge "popcloud:${BUCKET}/${PREFIX}"

Copy files up

shell

verified

# --checksum compares by hash rather than size and mtime, which is what you
# want whenever the source is another object store.
rclone copy ./public "popcloud:${BUCKET}/${PREFIX}" --checksum --progress

Verify a copy

shell

verified

# Compare both sides without transferring anything. Exit code 0 means every
# object matched by hash.
rclone check ./public "popcloud:${BUCKET}/${PREFIX}" --checksum --one-way

Copy from another provider

shell

not run in CI

# Streams provider to provider — nothing lands on the local disk. Define both
# remotes in rclone.conf first ([s3-old] for the source, [popcloud] for us).
rclone copy s3-old:legacy-bucket "popcloud:${BUCKET}" \
  --checksum \
  --transfers 16 \
  --checkers 32 \
  --s3-upload-concurrency 8 \
  --s3-chunk-size 32M \
  --stats 30s \
  --log-file migration.log

Verify the migration

shell

not run in CI

# Re-run after the copy. --one-way ignores anything already on the destination,
# so it answers exactly one question: did everything on the source arrive?
rclone check s3-old:legacy-bucket "popcloud:${BUCKET}" --checksum --one-way

Catch up before cutover

shell

not run in CI

# Cutover pass: only what changed since the bulk copy. Run it with writes to the
# source stopped, and it finishes in minutes rather than hours.
rclone sync s3-old:legacy-bucket "popcloud:${BUCKET}" \
  --checksum \
  --transfers 16 \
  --stats 30s

What to watch for

These apply to rclone specifically. The full list covers the platform.

Blocker

Keys containing . or .. path segments are rejected

You’ll see: 403 SignatureDoesNotMatch for a key with a dot segment in it, such as `./relative/../path.bin`, while every other key in the same run works.
Why: The URL is normalised somewhere between signing and verification, so the path that gets verified is not the path that was signed. This is a defect.
Instead: Normalise keys before uploading — collapse `.` and `..` segments yourself. This matters most when keys are derived from filesystem paths during a migration; a bulk copy will otherwise fail on a small, unpredictable subset of objects.

Friction

Buckets are not created over the S3 API

You’ll see: 405 MethodNotAllowed from CreateBucket or DeleteBucket — `aws s3 mb`, `mc mb`, or a tool provisioning its own bucket on first run.
Why: Bucket lifecycle belongs to the control plane, which also allocates the storage account, the data-plane key and the CORS rules that come with it. Letting the edge create buckets would create half of one.
Instead: Create buckets in the dashboard, or with `POST /v1/buckets` on the control-plane API. Everything object-level then works normally against that bucket.

Worth knowing

Only the STANDARD storage class exists

You’ll see: 501 NotImplemented when a request sets x-amz-storage-class to GLACIER, STANDARD_IA, INTELLIGENT_TIERING or similar.
Why: Storage economics are expressed as a bucket's tier (hot or cool), chosen when the bucket is created, rather than per object.
Instead: Leave the storage class unset. If your tool sets one by default, point it at STANDARD. To store colder data, create a cool-tier bucket and write there. RestoreObject does not exist either, because nothing is ever archived.

Worth knowing

The bucket listing omits some elements strict parsers expect

You’ll see: A tool fails to parse the bucket list, shows no owner, or sorts every bucket as if it were created just now.
Why: The bucket list is synthesized at the edge from the organisation's own buckets, rather than proxied. It currently ships without an XML namespace declaration or an Owner element, and stamps CreationDate with the time of the request.
Instead: Address buckets by name rather than discovering them from the listing, and do not rely on CreationDate. Every other listing operation — ListObjects, ListObjectsV2, ListObjectVersions, ListParts, ListMultipartUploads — is proxied and complete.

How this page is kept true

Every snippet above was extracted from examples/tools/rclone, a program that uploads, downloads, compares bytes, lists, presigns and cleans up after itself. It runs in CI against a live sandbox organisation via make docs-verify-tools. If it stops passing, this page is wrong and we treat that as a bug in the product.

A few snippets on this page are marked not run in CI — they need a browser, a second provider, or a configuration change the sandbox cannot make. Those are reviewed by hand.

Not yet run against the live sandbox — the runner is wired, the first verified run stamps this line.