matrixorigin / matrixorigin/matrixone

[Bug]: 4.2→main switch breaks catalog upgrade and rollback fails on unknown HAKeeper command

Open
#27,967 15 comments 0 reactions 1 assignee Claimed by @Ariznawlll View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

A TKE cluster whose shared data was created by the `4.2-dev` line cannot start CNs after all MatrixOne components are rolled forward to a current `main` image. During the bootstrap-upgrade path, both TP and AP CNs panic because the existing catalog does not contain `mo_catalog.mo_view_dependencies`.

This is not a query failure after startup: the CN reaches the upgrade framework, fails while checking/upgrading the existing catalog, panics, and enters `CrashLoopBackOff`.

## Environment and exact generations

- Warm Session / namespace: `mo-warm-33482153822-1`
- Dataset: `tpch_1g`, loaded once from the `4.2-dev` line
- Dataset anchor SHA: `8802a9865ff3379f82be258c38dcf660096a196a` (`4.2-dev`)
- Last known-good 4.2 generation: `60fb48fdc33fb7b9baee790597a6acd2ba2369e3`
- Last known-good runtime image ID: `ccr.ccs.tencentyun.com/matrixone-dev/matrixone@sha256:43ec879d80a24faed573a198ec08180512e38263298cf1f2d8decc15812da9a9`
- Failing main SHA: `592c680706b028f6e89365b6c105f8c32567e5bb`
- Failing main image digest: `sha256:8118866282834e6bf44407530ba60553d0c54526d5c789abee9513869b39d978`
- Workflow run: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33492159301
- Failing job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33492159301/job/99806435651

The Session reuses the same TN/log/shared object-storage data while changing the MatrixOne image. The rollout updated log service, TN, proxy, TP CN, and AP CN generations in place. The new main CNs then repeatedly crashed.

## Reproduction

1. Create a MatrixOne cluster with a `4.2-dev` image.
2. Load a normal dataset and retain its TN/log/shared object-storage state.
3. Update all MatrixOne components to main SHA `592c680706b028f6e89365b6c105f8c32567e5bb` / digest `sha256:811886...` without deleting the existing data.
4. Wait for the CNs to start.

Both TP and AP CNs enter `CrashLoopBackOff`.

## Relevant startup log

```text
start to check bootstrap state
mo already bootstrapped
start bootstrap upgrade
sql_executor exec: select rel_id from mo_catalog.mo_tables
where account_id=0 and reldatabase='mo_catalog'
and relname='mo_view_refresh' for update
internal sql executor error:
SQL parser error: table "mo_view_dependencies" does not exist
panic: SQL parser error: table "mo_view_dependencies" does not exist
```

The panic is raised from `main.startCNService.func1` (`cmd/mo-service/main.go:362`). The same error is reproducible on both the TP CN and an AP CN.

## Suspected compatibility boundary

`mo_view_dependencies` was added on main after the release-line divergence:

- `1f41696218068ad9dfd6f8cbe7fe576b2455b82e` — `refactor(view): add metadata regeneration foundation (#27267)`
- `dae2725bef5c81dcb9b5d2fab9a48afe37e2c760` — `refactor(view): add inactive metadata lifecycle (#27370)`

The failing main SHA contains those changes, while the retained 4.2 catalog predates the new system tables. The bootstrap-upgrade path appears to reference `mo_view_dependencies` before ensuring that the missing system table has been created/migrated.

## Expected behavior

For a supported upgrade from the current 4.2 release line to main:

- bootstrap upgrade creates/migrates newly required catalog tables in dependency order;
- CN startup completes without a panic;
- existing user data remains queryable.

If this cross-line transition is intentionally unsupported, MatrixOne should still detect the catalog-version incompatibility before a partial rolling update and return an explicit, actionable compatibility error. A raw CN startup panic and `CrashLoopBackOff` should not be the failure mode.

## Impact

- A release-line cluster cannot be rolled forward to main for validation while retaining data.
- Warm bisect/session workflows cannot compare a 4.2 candidate with main on the same loaded dataset.
- A partial component rollout leaves the cluster unavailable and requires an exact image rollback.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.