Migrate from Standard S3 Storage

CloudBase Cloud Storage is compatible with the S3 protocol. You can use rclone to migrate data from other S3-compatible storage (Supabase Storage, Alibaba Cloud OSS, Tencent Cloud COS, etc.) to CloudBase Cloud Storage.

This guide covers cloud storage file (objects in buckets) migration only. It does not cover database, authentication, cloud functions, or other resource migration.

Prerequisites: Install rclone

rclone is the core tool for this migration. Written in Go, it is a single binary with no external dependencies and supports data synchronization between all S3-compatible storage services.

Official site: https://rclone.org
GitHub: https://github.com/rclone/rclone

macOS
Linux
Windows

Recommended: install via Homebrew.

brew install rclone

Or download the binary manually:

curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip
unzip rclone-current-osx-amd64.zip
cd rclone-*-osx-amd64
sudo cp rclone /usr/local/bin/
sudo chown root:wheel /usr/local/bin/rclone
sudo chmod 755 /usr/local/bin/rclone

Recommended: use the official install script.

curl https://rclone.org/install.sh | sudo bash

Or use your system package manager:

# Debian / Ubuntu
sudo apt install rclone

# CentOS / RHEL
sudo yum install rclone

Recommended: install via Scoop.

scoop install rclone

Or use Chocolatey:

choco install rclone

Alternatively, download the Windows installer from the rclone downloads page, extract it, and place rclone.exe in your PATH.

Verify Installation

$ rclone version
rclone v1.74.4
- os/version: darwin 15.3 (64 bit)
- os/kernel: 24.3.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.26.5
- go/linking: dynamic
- go/tags: cmount

rclone remote configuration file location:

OS	Config file path
macOS / Linux	`~/.config/rclone/rclone.conf`
Windows	`%USERPROFILE%\.config\rclone\rclone.conf`

Step 1: Configure the CloudBase Target

The CloudBase remote configuration is the same regardless of the source.

Obtain CloudBase Credentials

Log in to Tencent Cloud Access Management Console to get your SecretId and SecretKey
Identify the COS bucket name and its Region in the CloudBase Cloud Storage Console

Configure the Remote

# Edit the config file directly
cat >> ~/.config/rclone/rclone.conf << 'EOF'
[cloudbase]
type = s3
provider = Other
access_key_id = <Your SecretId>
secret_access_key = <Your SecretKey>
endpoint = cos.ap-shanghai.myqcloud.com
region = ap-shanghai
force_path_style = false
EOF

💡 The [cloudbase] in the config block is the remote alias you assign to the CloudBase target. All subsequent rclone commands reference this endpoint via cloudbase:; the : is rclone's separator between remote and path. You can rename it to anything you like (e.g. [tcb]) as long as you keep it consistent throughout.

💡 To learn more about CloudBase Cloud Storage operations, see the Cloud Storage Overview.

⚠️ Important:

Region and endpoint must match. e.g., for Shanghai region use ap-shanghai, endpoint should be cos.ap-shanghai.myqcloud.com

force_path_style = false is mandatory: Tencent Cloud COS only supports Virtual Hosted-Style addressing. Omitting this will result in 403 PathStyleDomainForbidden.

Verify Connectivity

# List all buckets under CloudBase
rclone lsd cloudbase:

# List files in the target bucket (should be empty or contain only existing data)
rclone ls cloudbase:<target-bucket>

Step 2: Configure the Source S3 Remote

Select the configuration that matches your source storage type.

Migrate from Supabase Storage

Log in to Supabase Dashboard → target project → Settings → Storage
Enable S3 connection and generate S3 Access Key and S3 Secret Key
Note the project reference ID (the <project-ref> part of your Project URL)

cat >> ~/.config/rclone/rclone.conf << 'EOF'
[supabase-source]
type = s3
provider = Other
access_key_id = <Supabase S3 Access Key>
secret_access_key = <Supabase S3 Secret Key>
endpoint = https://<project-ref>.supabase.co/storage/v1/s3
region = <Source Region, e.g., ap-south-1>
EOF

# Verify connectivity
rclone lsd supabase-source:
rclone ls supabase-source:<source-bucket>
rclone size supabase-source:<source-bucket>

Migrate from Alibaba Cloud OSS

Log in to Alibaba Cloud RAM Console to get your AccessKey ID and AccessKey Secret
Confirm the bucket name, Region, and endpoint in the OSS Console

cat >> ~/.config/rclone/rclone.conf << 'EOF'
[oss-source]
type = s3
provider = Alibaba
access_key_id = <Alibaba Cloud AccessKey ID>
secret_access_key = <Alibaba Cloud AccessKey Secret>
endpoint = oss-<Region>.aliyuncs.com
region = <Region>
EOF

# Verify connectivity
rclone lsd oss-source:
rclone size oss-source:<source-bucket>

Migrate from Tencent Cloud COS

Same account, same region (e.g., both source and target in Guangzhou under the same account) — no need for a separate remote. Reuse cloudbase directly:

rclone copy cloudbase:<source-bucket> cloudbase:<target-bucket> --progress

Different account or different region — create a separate source remote:

cat >> ~/.config/rclone/rclone.conf << 'EOF'
[cos-source]
type = s3
provider = Other
access_key_id = <SecretId>
secret_access_key = <SecretKey>
endpoint = cos.<source-region>.myqcloud.com
region = <source-region>
force_path_style = false
EOF

# Verify connectivity
rclone lsd cos-source:
rclone size cos-source:<source-bucket>

Migrate from Other S3-Compatible Storage

AWS S3, MinIO, Qiniu Kodo, UpYun USS, and other S3-compatible storage services can all be accessed via rclone's S3 type:

cat >> ~/.config/rclone/rclone.conf << 'EOF'
[s3-source]
type = s3
provider = <Use "AWS" for AWS, "Minio" for self-hosted MinIO, "Other" for others>
access_key_id = <Access Key>
secret_access_key = <Secret Key>
endpoint = <S3 Endpoint>
region = <Region>
EOF

For more provider types, refer to the rclone S3 documentation.

Step 3: Execute the Migration

We recommend executing in three steps: dry-run preview first, then actual migration, then integrity verification.

# Step 1: Dry run, preview files to be transferred
rclone copy <source-remote>:<source-bucket> cloudbase:<target-bucket> \
  --dry-run \
  --progress \
  --transfers 4

# Step 2: Execute the actual migration
rclone copy <source-remote>:<source-bucket> cloudbase:<target-bucket> \
  --progress \
  --transfers 4 \
  --checkers 8 \
  --retries 3

# Step 3: One-way integrity check
rclone check <source-remote>:<source-bucket> cloudbase:<target-bucket> \
  --one-way \
  --progress

Performance Tuning Suggestions

Parameter	Purpose	Recommended Value
`--transfers N`	Concurrent file transfers	8-16 for same region, 4 for cross-cloud
`--checkers N`	Concurrent checkers	Usually 2× transfers
`--retries N`	Retry count on failure	3
`--bwlimit X`	Bandwidth limit	`10M` recommended for cross-cloud to avoid saturation
`--fast-list`	Reduce API calls	Recommended for large buckets

# Same-region high-speed transfer
rclone copy <source>:<source-bucket> cloudbase:<target-bucket> \
  --transfers 16 --checkers 32 --fast-list --progress

# Cross-cloud rate-limited transfer
rclone copy <source>:<source-bucket> cloudbase:<target-bucket> \
  --transfers 4 --bwlimit 10M --retries 3 --progress

About Verification Results

If you see N hashes could not be checked when running rclone check --one-way, this is not an error. Different S3 providers use different ETag/MD5 calculation methods, so rclone cannot directly compare checksums and will automatically fall back to verifying by file name + file size. As long as the final output shows 0 differences + N matching files, the files are fully consistent.

Resume from Interruption

rclone's copy command natively supports resume: re-running the same command after an interruption automatically skips completed files and continues with the rest.

# After Ctrl+C during transfer, simply rerun the same command
rclone copy <source>:<source-bucket> cloudbase:<target-bucket> --progress

rclone Command Reference

Browsing and Viewing

rclone lsd remote:              # List all buckets
rclone ls remote:bucket         # Recursively list files (with size)
rclone lsl remote:bucket        # List files (with size + modification time)
rclone tree remote:bucket       # Display directory structure as a tree
rclone size remote:bucket       # Count files and total size

Transfer: copy vs move vs sync

Command	Purpose	Source files	Extra files in target
`copy`	Copy	Kept	Kept
`move`	Move	Deleted	Kept
`sync`	Sync (mirror)	Kept	Deleted

sync makes the target bucket identical to the source. Extra files in the target will be deleted. Always use --dry-run first.

Verification and Comparison

# Checksum verification
rclone check remote:src remote:dst

# One-way check (source → target only)
rclone check remote:src remote:dst --one-way

# Verify by file size (when cross-cloud checksum is unavailable)
rclone check remote:src remote:dst --size-only --one-way

Filter Transfers

# Only transfer images
rclone copy remote:src remote:dst --include "*.{jpg,png,gif}"

# Exclude temporary files
rclone copy remote:src remote:dst --exclude "*.tmp" --exclude ".DS_Store"

# Filter by size
rclone copy remote:src remote:dst --min-size 1M

FAQ

Error	Likely Cause	Solution
`403 Forbidden`	Incorrect credentials or insufficient permissions	Check SecretId/SecretKey, confirm read/write access to bucket
`403 PathStyleDomainForbidden`	rclone defaults to Path-Style	Add `force_path_style = false` to the CloudBase remote config
`404 Not Found`	Incorrect bucket name	Confirm bucket name is correct, endpoint matches Region
`SignatureDoesNotMatch`	Signature error	SecretKey may have extra spaces/newlines, re-copy it
`connection refused`	Network unreachable	Check firewall/proxy, confirm endpoint is publicly accessible
`slow transfer`	Cross-cloud public network is slow	Increase `--transfers` concurrency, or remove `--bwlimit`

Debug Commands

# View rclone configuration
rclone config show

# Run in debug mode (outputs detailed HTTP request logs)
rclone copy source:bucket cloudbase:bucket --progress -vv

# Quick comparison by file size (no checksum verification)
rclone check source:bucket cloudbase:bucket --one-way --size-only

Prerequisites: Install rclone​

Verify Installation​

Step 1: Configure the CloudBase Target​

Obtain CloudBase Credentials​

Configure the Remote​

Verify Connectivity​

Step 2: Configure the Source S3 Remote​

Migrate from Supabase Storage​

Migrate from Alibaba Cloud OSS​

Migrate from Tencent Cloud COS​

Migrate from Other S3-Compatible Storage​

Step 3: Execute the Migration​

Performance Tuning Suggestions​

About Verification Results​

Resume from Interruption​

rclone Command Reference​

Browsing and Viewing​

Transfer: copy vs move vs sync​

Verification and Comparison​

Filter Transfers​

FAQ​

Debug Commands​

Next Steps​