Migrating Existing Files

This guide explains how to migrate files uploaded with legacy SDKs into a cloud storage bucket so they regain bucket ownership.

Who this applies to

You used a legacy SDK (JS SDK V2 / old Node SDK / old wx-server-sdk) to upload files in a PG environment, and the top of the "Cloud Storage - Storage Management" page in the console shows "Files uploaded by legacy SDKs do not belong to any bucket".

Before You Start

What "existing files" means: files uploaded with a legacy SDK that currently belong to no bucket. Their bytes are already in the environment's COS bucket — only the "ownership" step is missing. As a result, they are invisible when browsing by bucket in the console, and they are not governed by bucket permissions or policies.

Common questions about the migration itself:

Question	Answer
Will it affect production traffic?	No. The migration only reads source files and writes new objects to the target bucket. Existing files are neither modified nor deleted (source files are kept by default)
Do I need to stop the service?	No, it can be done online
How long does it take?	Depends on file count and total size. The script defaults to concurrency 4 (adjustable with `--concurrency`) and supports resuming — rerunning after an interruption continues where it left off
Will it incur extra costs?	During migration, the same data briefly exists twice (the original source file and the bucket copy). After confirming everything is correct, run `cleanup` to remove the source copy and return to a single copy
Do I need to change paths in business code?	No. The migration does not change object names (keys); it only adds bucket ownership
Do I still need to migrate after upgrading the SDK?	Yes. The upgrade only routes newly uploaded files through the bucket path; already-uploaded files are not migrated automatically

Recommendation

First run the full flow in a test environment with a small data range (--source-prefix), then migrate production in full.

Two Things You Need to Do

What you do	What it solves	If you skip it
Upgrade the SDK	Routes "newly uploaded files" through the bucket path	Once the direct-upload path is disabled, upload requests are rejected outright
Migrate existing files (this guide)	Restores ownership for "files already uploaded"	Files stay invisible by bucket in the console and are not governed by bucket policies

Upgrade paths: JS SDK V3 (recommended) | Node SDK V4.1 | wx-server-sdk (Mini Program)

Migration Overview

Check  ──►  Identify  ──►  Migrate  ──►  Verify  ──►  Close out
tools/creds/  list files    write to      confirm all     optionally clean
source conn.  to migrate    target bucket  owned & applied  source, hide banner

The essence of the migration: the file bytes are already in the environment's COS bucket; what is missing is ownership. The migration writes them into the specified bucket through the official gateway, so that byte placement and metadata registration are both satisfied.

A migration script accompanies this guide. Download it first: Download pg-bucket-migration.zip (includes the migration script and a configuration template; unzip and run, Node.js ≥ 18, zero npm dependencies).

All commands below are run in the unzipped pg-bucket-migration directory.

Prerequisites

Confirm the target bucket exists. Create or select the target bucket in the console under "Cloud Storage - Storage Management" (bucket1 is used as an example below):
```
tcb storage buckets list -e <envId>
```
Log in to the tcb CLI (the migration uses the CLI backend by default):
```
tcb login          # Recommended: interactive authorization, secrets never hit the command line
```
Note
For non-interactive login (CI scenarios), you can use tcb login --apiKeyId <SecretId> --apiKey <SecretKey>. This exposes secrets in your shell history and process list, so prefer temporary credentials and disable them promptly after the migration.

Install rclone (used to enumerate and pull source files):

# macOS
brew install rclone
# Linux
curl https://rclone.org/install.sh | sudo bash
# Windows: scoop install rclone

Fill in the configuration:

unzip pg-bucket-migration.zip && cd pg-bucket-migration
cp .env.example .env
vim .env          # set CB_ENV_ID / CB_TARGET_BUCKET / COS credentials or CB_SOURCE

Permissions required: read-only access to the source COS bucket and read-write access to the target bucket.

Secret safety

.env contains plaintext secrets. It is already listed in .gitignore — do not commit it to a repository or share it externally.

Step 1: Pre-migration Check

First confirm that tools, credentials, source connectivity, and APIs all work, so you do not discover problems halfway through:

node migrate-orphaned-to-bucket.mjs check

── Check result ───────────────────────────
  rclone              : available
  Source accessible   : cbstorage-src:<bucket-APPID> (1,247 objects, 2.31 GB)
  envId               : my-env-id
  Target bucket       : bucket1
  Storage backend     : tcb CLI (available, bucket: bucket1)
  Bucket object list  : OK (currently 402 objects)
───────────────────────────────────────────

If anything is abnormal, fix it as prompted before continuing.

Step 2: Identify Files to Migrate

The script subtracts "files that already have ownership" from "all source files" to produce the migration list. This step is read-only and writes no data, so it can be run repeatedly.

node migrate-orphaned-to-bucket.mjs plan

→ Enumerating existing source files (COS)…
  Source files: 1,247, total 2.31 GB
→ Listing objects already in buckets (pg bucket "bucket1")…
  Already in bucket: 402

── Migration preview ──────────────────────────
  To migrate (no ownership): 845, 1.86 GB
  Already in bucket (skipped): 402
  Already-owned physical objects: 402 (owned by other buckets, skipped automatically)
  Manifest written to: .pg-bucket-migration/manifest.json
───────────────────────────────────────────────

"Already-owned physical objects" are objects that already belong to a bucket (their COS path carries a bucket ID prefix). The script identifies and skips them automatically — no manual handling needed.

Review the list before deciding how to migrate. Common parameters:

Scenario	How to handle
Migrate only one directory	`--source-prefix uploads/`
Exclude certain files	`--exclude "tmp/**"`
Source paths carry an extra prefix to strip	`--strip-prefix legacy/`
Handle a few large files separately	`--max-file-size 104857600` (skip files >100MB)

Step 3: Run the Migration

node migrate-orphaned-to-bucket.mjs migrate --yes

The script processes files one by one: pull the source file → write to the target bucket through the official gateway → record the result → continue.

Will migrate 845 files (1.86 GB) into bucket "bucket1"
Strategy: reupload
  ✓ [1/845] uploads/avatar-001.jpg (1.24 MB)
  ✓ [2/845] docs/manual.pdf (3.80 MB)
  ✓ [3/845] images/banner.png (2.10 MB)
  …
── Migration result ───────────────────────
  Succeeded: 845 / 845     Failed: 0
───────────────────────────────────────────

Features:

Resumable: after an interruption or partial failure, rerun the same command and successfully migrated files are skipped automatically
Idempotent: if an object with the same name already exists in the target bucket, it is overwritten; rerunning does not fail
Default concurrency 4, adjustable with --concurrency 8
Source files kept by default (see Optional: Reclaim Source Space)

The migration does not change object names (keys). A file named uploads/avatar-001.jpg keeps that name after joining the bucket, so no business code changes are needed.

Step 4: Verify the Result

node migrate-orphaned-to-bucket.mjs verify

── Verification result ────────────────────
  Files in manifest   : 845
  Objects in bucket   : 1,247
  Missing (no owner)  : 0
  Size mismatches     : 0
───────────────────────────────────────────

✅ All files to migrate now belong to the bucket.

After verification passes, do a final check in the console under "Cloud Storage - Storage Management":

The files are visible under the target bucket
File sizes and paths match pre-migration values
Bucket access permissions, policies, file size limits, and allowed file types now apply to them (try a read/write per your policies)
Business reads and writes have been validated in a test environment

If anything is still missing or mismatched, rerun migrate (successful items are skipped automatically) and then verify again.

Once all files to migrate belong to the bucket, close out the banner at the top of the console's "Storage Management" page:

node migrate-orphaned-to-bucket.mjs close-banner

The script prints the exact steps: set the environment flag postgresql_orphaned_storage to false as prompted.

Note

Only do this when the environment truly has no orphaned files left; otherwise you may dismiss the banner for other users by mistake. Run plan first and confirm "to migrate" is 0.

Optional: Reclaim Source Space

The migration keeps source files by default so you can roll back at any time. However, the same data occupies space twice, so cleaning up after confirming normal operation is recommended:

# Preview which files would be deleted
node migrate-orphaned-to-bucket.mjs cleanup --dry-run

# Run it after confirming
node migrate-orphaned-to-bucket.mjs cleanup

── Source cleanup preview ─────────────────
  Cleanable (same-name object exists in bucket): 845, 1.86 GB
    - uploads/avatar-001.jpg
    - docs/manual.pdf
    …
  Skipped (no same-name object in bucket, left untouched): 0
───────────────────────────────────────────

Safety: cleanup deletes a source file only when it is recorded as successfully migrated and an object with the same name indeed exists in the target bucket; everything else is skipped. It runs item by item and records immediately, so it can be safely rerun after an interruption.

You can also clean up during the migration: node migrate-orphaned-to-bucket.mjs migrate --delete-source.

FAQ

Migration

Question	Answer
I upgraded the SDK — do I still need to migrate?	Yes. The upgrade only handles newly uploaded files; already-uploaded files are not migrated automatically
Do I need to change code after migrating?	No. Object names are unchanged; only bucket ownership is added
What if there are many files (>10GB)?	Migrate in batches by directory (`--source-prefix`), or use the console's bulk import tool
What if the migration is interrupted?	Rerun the same `migrate` command; successfully migrated files are skipped
Can I roll back after migrating?	Yes. Source files are kept by default; delete the objects migrated into the target bucket to roll back

Error handling

Symptom	Cause	Handling
`rclone not found`	Not installed or not in PATH	`brew install rclone`, or point `RCLONE_BIN` at the binary
`403 PathStyleDomainForbidden`	COS only supports Virtual Hosted-Style addressing	Make sure the rclone config includes `force_path_style=false`
`STORAGE_KEY_ALREADY_EXISTS`	An object with the same name exists in the target bucket and the API rejects overwrites by default	The script handles this automatically; for manual operations add `--upsert`
`STORAGE_BUCKET_NOT_FOUND`	The target bucket does not exist	Create the bucket in the console first
`STORAGE_PERMISSION_DENIED`	The bucket's RLS policy does not allow the operation	Configure the bucket RLS policy and retry
The migration list looks far too large	Non-business files are mixed into the source	Exclude them with `--exclude`
Some file uploads fail	Network jitter or an oversized file	Rerun `migrate` to retry automatically; or lower concurrency and raise `--retries`

Security and Rollback

Validate on a small scale first: pick one subdirectory with --source-prefix, complete the full flow, then migrate everything.
Source files are not deleted by default: source files stay as they are, so you can roll back at any time.
Least privilege: grant read-only access to the source bucket and read-write access to the target bucket only; disable or delete the credentials promptly after the migration.
How to roll back: the migration does not damage source data. To roll back, delete the objects migrated into the target bucket (tcb storage objects rm <key> --bucket <bucket>).

Appendix A: Script Package Contents

The downloaded pg-bucket-migration.zip unzips into a directory with the following layout:

pg-bucket-migration/
├── migrate-orphaned-to-bucket.mjs
├── .env.example
├── README.md
├── .gitignore
└── 存量文件迁移指引.md

File	Purpose	Required
`migrate-orphaned-to-bucket.mjs`	Main migration script containing all check / identify / migrate / verify / close-out logic, with zero npm dependencies	Yes
`.env.example`	Configuration template; copy to `.env` and fill in the environment and credentials to avoid hand-built variables	Recommended
`README.md`	Directory description, dependency requirements, and a command list — visible right after unzipping	Recommended
`.gitignore`	Ignores `.env` and the state directory to keep secrets out of your repository	Recommended
`存量文件迁移指引.md`	This guide, available offline	Optional

Runtime requirements: Node.js ≥ 18 and rclone; the tcb CLI is also required when using the default CLI backend. See Prerequisites for environment setup.

After the first run, the script creates a state directory .pg-bucket-migration/ in the same directory (rename it with --state-dir or CB_MIGRATION_DIR):

.pg-bucket-migration/
├── manifest.json
├── state.json
├── report.json
└── verify.json

File	Produced by	Purpose
`manifest.json`	`plan`	List of files to migrate and skip statistics
`state.json`	`migrate`	Per-file results; the basis for resuming
`report.json`	`migrate`	Summary of successes / failures for this run, plus failure details
`verify.json`	`verify`	Verification conclusion: missing count, size consistency, pass or fail

The state directory only records migration results and contains no secrets, so it can be deleted at any time. It is regenerated on the next run, at the cost of losing resume capability and the migration audit trail.

Appendix B: Manual Migration Without the Script

When there are very few files (single digits), or running a script is not convenient, you can do it entirely by hand.

# 1. Inspect the target bucket and its objects
tcb storage buckets list -e <envId>
tcb storage objects list --bucket bucket1 -e <envId>

# 2. Inspect source files (rclone connects directly to the environment's COS bucket; note force_path_style = false)
rclone ls cbstorage-src:<bucket-APPID>

# 3. Write each file into the target bucket (--upsert allows overwriting a same-name object)
rclone copyto "cbstorage-src:<bucket-APPID>/uploads/avatar-001.jpg" ./tmp.jpg
tcb storage objects upload ./tmp.jpg "uploads/avatar-001.jpg" --bucket bucket1 -e <envId> --upsert

Note

The manual approach has no difference detection, resume capability, or verification, and errors become likely once there are many files. It is not recommended for bulk migrations.

Before You Start​

Two Things You Need to Do​

Migration Overview​

Prerequisites​

Step 1: Pre-migration Check​

Step 2: Identify Files to Migrate​

Step 3: Run the Migration​

Step 4: Verify the Result​

Step 5: Dismiss the Console Banner​

Optional: Reclaim Source Space​

FAQ​

Migration​

Error handling​

Security and Rollback​

Appendix A: Script Package Contents​

Appendix B: Manual Migration Without the Script​