matrixorigin / matrixorigin/matrixone
[Bug]: DATA BRANCH DIFF fails after GC and restart
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
`DATA BRANCH DIFF` cannot read a live branch after storage GC and a service restart. A minimal one-edge branch is sufficient: the pre-restart DIFF is correct, while the first post-restart DIFF fails during history lookup even though both tables and the branch protection record remain present.
## Environment
- Branch: `main`
- Commit: `dd040ad6913e9b86f592b574e5fffb610003f72a`
- Deployment: local single-process topology (1 CN, 1 TN, 1 Log Service), local filesystem storage
- Test date: 2026-08-22
## Steps to reproduce
```sql
create database db_gc_restart;
use db_gc_restart;
create table root(id int primary key, v int);
insert into root values (1,1),(2,2),(3,3),(4,4);
data branch create table leaf from root;
update root set v=101 where id=1;
delete from root where id=2;
update leaf set v=303 where id=3;
insert into leaf values(5,505);
-- Before lifecycle actions this returns four changes.
data branch diff leaf against root output count;
select mo_ctl('dn','flush','db_gc_restart.root');
select mo_ctl('dn','flush','db_gc_restart.leaf');
select mo_ctl('dn','globalcheckpoint','');
select mo_ctl('dn','diskcleaner','force_gc');
-- Still returns four before restart.
data branch diff leaf against root output count;
-- Gracefully restart this local MO service, then reconnect.
data branch diff leaf against root output count;
```
## Actual behavior
The post-restart DIFF fails with:
```text
HY000 / 20101: internal error: cannot find database (db_gc_restart) created_time at snapshot
```
The failed read does not alter visible data: in the expanded chain matrix, the root table retained 3 rows, the leaf retained 5 rows, and 2 branch-protection records remained present after every failure.
## Expected behavior
GC and restart must not invalidate history required by a live Data Branch. The post-restart DIFF should return the same four changes as before restart.
## Stability and controls
- Minimal one-edge reproducer with GC + restart: 3/3 failed on latest `main`.
- Expanded `root -> middle -> leaf` lifecycle matrix with GC + restart: 10/10 failed; each run passed the pre-restart DIFF.
- Same expanded topology with restart but without GC: 3/3 passed, including exact DIFF results and subsequent branch-history reclamation.
- Failure checks: both table contents and branch protection metadata remained intact after each error.
- No branch-table `mergeobjects` is required. This distinguishes the issue from #27451, which requires branch-table compaction and fails with a missing object error before restart.
## Evidence
- `DataBranchGcRestartDirectProbe.java`: minimal 3-run matrix.
- `DataBranchProtectRecoveryProbe.java`: 10-run chain matrix plus 3-run restart-only control.
## Code analysis
Confirmed failure site: `getDatabaseCreatedTimeLowerBoundByPK` in `pkg/frontend/data_branch.go` scans `mo_database` at the branch history timestamp and raises this error when it cannot find the database creation timestamp.
Hypothesis: after GC/restart, the snapshot catalog scan used to derive the collection lower bound no longer sees the still-live database's historical creation record. The helper was introduced by #24371 while optimizing arbitrary-DAG collection ranges; the fallback or retention path should preserve a usable lower bound for a live branch.
## Regression coverage
After the fix, add a deterministic case under `motr/suites/12_git4data` / `test/distributed/cases/git4data/branch/diff/` that creates a one-edge branch, advances flush/checkpoint/GC, restarts the service in the appropriate recovery workflow, and asserts exact post-restart DIFF rows plus table/protection state.
## Related
- #24010 — earlier GC/restart Data Branch DIFF repair; this is a new recurrence with a different failing path.
- #27451 — distinct branch-table-compaction/GC history loss without restart.
Contributor guide
Assessment
This issue has not been assessed yet.