matrixorigin / matrixorigin/matrixone
[Bug]: 4.2→main switch breaks catalog upgrade and rollback fails on unknown HAKeeper command
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
A TKE cluster whose shared data was created by the `4.2-dev` line cannot start CNs after all MatrixOne components are rolled forward to a current `main` image. During the bootstrap-upgrade path, both TP and AP CNs panic because the existing catalog does not contain `mo_catalog.mo_view_dependencies`.
This is not a query failure after startup: the CN reaches the upgrade framework, fails while checking/upgrading the existing catalog, panics, and enters `CrashLoopBackOff`.
## Environment and exact generations
- Warm Session / namespace: `mo-warm-33482153822-1`
- Dataset: `tpch_1g`, loaded once from the `4.2-dev` line
- Dataset anchor SHA: `8802a9865ff3379f82be258c38dcf660096a196a` (`4.2-dev`)
- Last known-good 4.2 generation: `60fb48fdc33fb7b9baee790597a6acd2ba2369e3`
- Last known-good runtime image ID: `ccr.ccs.tencentyun.com/matrixone-dev/matrixone@sha256:43ec879d80a24faed573a198ec08180512e38263298cf1f2d8decc15812da9a9`
- Failing main SHA: `592c680706b028f6e89365b6c105f8c32567e5bb`
- Failing main image digest: `sha256:8118866282834e6bf44407530ba60553d0c54526d5c789abee9513869b39d978`
- Workflow run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33492159301
- Failing job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33492159301/job/99806435651
The Session reuses the same TN/log/shared object-storage data while changing the MatrixOne image. The rollout updated log service, TN, proxy, TP CN, and AP CN generations in place. The new main CNs then repeatedly crashed.
## Reproduction
1. Create a MatrixOne cluster with a `4.2-dev` image.
2. Load a normal dataset and retain its TN/log/shared object-storage state.
3. Update all MatrixOne components to main SHA `592c680706b028f6e89365b6c105f8c32567e5bb` / digest `sha256:811886...` without deleting the existing data.
4. Wait for the CNs to start.
Both TP and AP CNs enter `CrashLoopBackOff`.
## Relevant startup log
```text
start to check bootstrap state
mo already bootstrapped
start bootstrap upgrade
sql_executor exec: select rel_id from mo_catalog.mo_tables
where account_id=0 and reldatabase='mo_catalog'
and relname='mo_view_refresh' for update
internal sql executor error:
SQL parser error: table "mo_view_dependencies" does not exist
panic: SQL parser error: table "mo_view_dependencies" does not exist
```
The panic is raised from `main.startCNService.func1` (`cmd/mo-service/main.go:362`). The same error is reproducible on both the TP CN and an AP CN.
## Suspected compatibility boundary
`mo_view_dependencies` was added on main after the release-line divergence:
- `1f41696218068ad9dfd6f8cbe7fe576b2455b82e` — `refactor(view): add metadata regeneration foundation (#27267)`
- `dae2725bef5c81dcb9b5d2fab9a48afe37e2c760` — `refactor(view): add inactive metadata lifecycle (#27370)`
The failing main SHA contains those changes, while the retained 4.2 catalog predates the new system tables. The bootstrap-upgrade path appears to reference `mo_view_dependencies` before ensuring that the missing system table has been created/migrated.
## Expected behavior
For a supported upgrade from the current 4.2 release line to main:
- bootstrap upgrade creates/migrates newly required catalog tables in dependency order;
- CN startup completes without a panic;
- existing user data remains queryable.
If this cross-line transition is intentionally unsupported, MatrixOne should still detect the catalog-version incompatibility before a partial rolling update and return an explicit, actionable compatibility error. A raw CN startup panic and `CrashLoopBackOff` should not be the failure mode.
## Impact
- A release-line cluster cannot be rolled forward to main for validation while retaining data.
- Warm bisect/session workflows cannot compare a 4.2 candidate with main on the same loaded dataset.
- A partial component rollout leaves the cluster unavailable and requires an exact image rollback.
Contributor guide
Assessment
This issue has not been assessed yet.