Releasing and Canarying Environment Branches
This document describes how to release changes to the production environment after environment branch development is complete, and how to perform online canary releases to ensure that issues in new versions only affect a small number of users and can be rolled back quickly.
Release Workflow
| Stage | Actions | Features Used |
|---|---|---|
| Develop and verify on a branch | Verify changes using data in the branch | Environment branches |
| Release to production | Apply verified changes sequentially to the production environment | Migration SQL, function deployment, and version release |
| Online canary release | Direct a portion of traffic to the new version first, then roll out to 100% after verification | Cloud function version traffic; self-hosted routing layer if needed |
Developing and Verifying on a Branch
A branch is an isolated environment cloned from the production environment, featuring its own unique environment ID, database replica, HTTP gateway, and access domain. You can verify changes and run tests using the data in the branch. For the capabilities available during the Private Alpha stage, see Recommendations for the Private Alpha Stage.
Although you can forward some user requests to the branch address using your own entry layer, branches differ from the production environment in the following ways:
- Data: The branch database is a copy created at a specific point in time. Data written to the branch will not sync back to the production environment.
- Login Session: User tokens issued by the production environment are invalid on the branch, and vice versa. Users must log in again on the branch.
- Configurations: Configurations such as HTTP gateway routes and login methods must be individually confirmed on the branch.
This method is only suitable for internal staff or beta users who can accept isolated data and accounts. It is not recommended for production canary releases. For online canary releases, perform them within the production environment (see Online Canary Releases); for forwarding methods, see Self-Hosted Routing Layer.
Releasing to Production
- Manage changes as files: Write database changes as migration SQL, and place function code and configurations under version control. Avoid manual modifications in the console or SQL window, as they cannot be accurately replayed in the production environment.
- Verify on the branch: Use the branch data to complete testing. For the verification scope available during the Private Alpha stage, see below.
- Review: Confirm change details through code reviews (such as Pull Requests).
- Release sequentially to production: Execute database migrations first, then deploy functions and release the new version, and finally roll out traffic gradually using version traffic.
Planned: Diff and Merge
We plan to provide branch merging capabilities in the future. This will allow you to compare differences between the branch and the production environment (referencing Supabase pg-delta for database schema diffs) and merge them into the production environment after confirmation. Data in the branch will not be merged.
Recommendations for the Private Alpha Stage
Environment branches are currently in Private Alpha. We provide APIs to create, view, and delete branches, as well as access branch data interfaces using branch temporary keys. Based on current capabilities, we recommend:
- Perform schema changes in production: Modifying table schemas on a branch is currently not supported. Write schema changes as migration SQL and execute them in production in a compatible-first, clean-up-later manner. If a branch requires specific tables, create those tables in production first before creating the branch.
- Verify data reads and writes on the branch: Use branch temporary keys to read and write data in the branch. This allows you to verify query and data processing logic, or let AI Agents operate without affecting production data.
Online Canary Releases
During a canary release, both the old and new versions must read and write the same production data. Therefore, the canary release must be performed inside the production environment.
Cloud Functions: Using Version Traffic
- Release a function version and direct 100% of online traffic to it. After this, deploying new code will only update
$LATESTwithout affecting online traffic. - Deploy the new code and release a new version.
- Allocate a small portion of traffic to the new version, monitor its performance, and gradually increase the percentage to 100%.
- If issues occur, roll back by setting the traffic for the old version back to 100%.
The behavior of version traffic allocation is as follows:
| Scenario | Behavior |
|---|---|
| Route has "Authentication" enabled | Requests from the same user are pinned to the same version |
| Route has "Authentication" disabled | Each request is randomly assigned to a version based on the allocation ratio |
| Gradually increasing the ratio of the new version | Users already routed to the new version will not be switched back to the old version |
| Multiple functions undergoing canary release simultaneously | With the same allocation ratio, the same user will route to the same version across different functions |
| Rollback | Changes to traffic allocation ratios typically take effect within tens of seconds |
To maintain version consistency per user, enable Authentication for the corresponding route. For configuration details, see Cloud Function Canary Releases.
Databases: Compatible First, Clean Up Later
Because both old and new versions share the same database, database changes take effect for both versions simultaneously. Database changes cannot be gradually rolled out via canary releases, nor can they be rolled back by switching traffic. We recommend a two-phase release:
- Compatible first: Only add tables, columns, or indexes, ensuring that functions of both the old and new versions run normally.
- Clean up later: Once the new version is fully stable and the old version no longer receives traffic, delete the obsolete columns or tables.
If issues arise, only roll back the function version; do not roll back the database. Fix issues by applying a new migration forward.
Frontend
Static website hosting does not support distributing frontend assets by version or user. If frontend issues occur, redeploying the previous build will restore service. For frontend canary releases, you can deploy the old and new builds to different paths and have a self-hosted routing layer return them based on rules.
Self-Hosted Routing Layer
For requirements beyond the scope of cloud function version traffic, you can add a self-hosted routing layer (such as CDN rules, edge functions, BFF, or Nginx) in front of CloudBase.
The HTTP gateway of each environment can only route requests to resources within that specific environment; the production gateway cannot forward requests to a branch environment. If you need to access both environments, the routing layer should be placed in front of the HTTP gateways of both environments to forward requests to the corresponding environment domains based on your rules.
┌─ Production Domain → Prod HTTP Gateway ─┬─ /api-stable → Function Version N
Client → Routing Layer ───┤ └─ /api-canary → Function Version N+1
│ (Shared Production DB)
└─ Branch Domain ────→ Branch HTTP Gateway ──→ Functions in Branch
(Internal/Beta Users Only)
| Requirement | Approach |
|---|---|
| Frontend canary | Deploy old and new frontend builds to different paths or sites, and let the routing layer return them based on rules |
| Canary by tenant or whitelist | Create routes for the old and new versions in the production HTTP gateway, binding them to specific function versions, and let the routing layer forward requests based on tenant or whitelist |
| Branch access for internal/beta users | Let the routing layer forward requests from specified users to the branch domain, subject to the limitations in Branches do not handle production traffic |
When performing a canary release by tenant or whitelist, on the CloudBase side, you only need to create a new route for the canary version in the production environment's HTTP gateway and bind it to the designated function version:
When designing the routing layer, keep the following in mind:
- Use only one split point: The downstream routes of the routing layer should bind to fixed versions. Do not overlay function version traffic on top of them.
- Use trusted identifiers: Split traffic based on verified user IDs or tenant IDs. Do not trust client-reported identifiers.
- Prioritize rollback switches: The switch to roll back to the old version must have a higher priority than other rules, such as whitelists or hardcoded routing.
- Do not retry across versions: If a request to the new version fails, return the error directly. Do not automatically retry the request on the old version.