matrixorigin / matrixorigin/matrixone

[Bug]: DATA BRANCH DIFF fails after GC and restart

Open
#27,465 9 comments 0 reactions 1 assignee Claimed by @gouhongshen View on GitHub
ai-easy deferred kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

`DATA BRANCH DIFF` cannot read a live branch after storage GC and a service restart. A minimal one-edge branch is sufficient: the pre-restart DIFF is correct, while the first post-restart DIFF fails during history lookup even though both tables and the branch protection record remain present.

## Environment

- Branch: `main`
- Commit: `dd040ad6913e9b86f592b574e5fffb610003f72a`
- Deployment: local single-process topology (1 CN, 1 TN, 1 Log Service), local filesystem storage
- Test date: 2026-08-22

## Steps to reproduce

```sql
create database db_gc_restart;
use db_gc_restart;
create table root(id int primary key, v int);
insert into root values (1,1),(2,2),(3,3),(4,4);

data branch create table leaf from root;
update root set v=101 where id=1;
delete from root where id=2;
update leaf set v=303 where id=3;
insert into leaf values(5,505);

-- Before lifecycle actions this returns four changes.
data branch diff leaf against root output count;

select mo_ctl('dn','flush','db_gc_restart.root');
select mo_ctl('dn','flush','db_gc_restart.leaf');
select mo_ctl('dn','globalcheckpoint','');
select mo_ctl('dn','diskcleaner','force_gc');

-- Still returns four before restart.
data branch diff leaf against root output count;

-- Gracefully restart this local MO service, then reconnect.
data branch diff leaf against root output count;
```

## Actual behavior

The post-restart DIFF fails with:

```text
HY000 / 20101: internal error: cannot find database (db_gc_restart) created_time at snapshot
```

The failed read does not alter visible data: in the expanded chain matrix, the root table retained 3 rows, the leaf retained 5 rows, and 2 branch-protection records remained present after every failure.

## Expected behavior

GC and restart must not invalidate history required by a live Data Branch. The post-restart DIFF should return the same four changes as before restart.

## Stability and controls

- Minimal one-edge reproducer with GC + restart: 3/3 failed on latest `main`.
- Expanded `root -> middle -> leaf` lifecycle matrix with GC + restart: 10/10 failed; each run passed the pre-restart DIFF.
- Same expanded topology with restart but without GC: 3/3 passed, including exact DIFF results and subsequent branch-history reclamation.
- Failure checks: both table contents and branch protection metadata remained intact after each error.
- No branch-table `mergeobjects` is required. This distinguishes the issue from #27451, which requires branch-table compaction and fails with a missing object error before restart.

## Evidence

- `DataBranchGcRestartDirectProbe.java`: minimal 3-run matrix.
- `DataBranchProtectRecoveryProbe.java`: 10-run chain matrix plus 3-run restart-only control.

## Code analysis

Confirmed failure site: `getDatabaseCreatedTimeLowerBoundByPK` in `pkg/frontend/data_branch.go` scans `mo_database` at the branch history timestamp and raises this error when it cannot find the database creation timestamp.

Hypothesis: after GC/restart, the snapshot catalog scan used to derive the collection lower bound no longer sees the still-live database's historical creation record. The helper was introduced by #24371 while optimizing arbitrary-DAG collection ranges; the fallback or retention path should preserve a usable lower bound for a live branch.

## Regression coverage

After the fix, add a deterministic case under `motr/suites/12_git4data` / `test/distributed/cases/git4data/branch/diff/` that creates a one-edge branch, advances flush/checkpoint/GC, restarts the service in the appropriate recovery workflow, and asserts exact post-restart DIFF rows plus table/protection state.

## Related

- #24010 — earlier GC/restart Data Branch DIFF repair; this is a new recurrence with a different failing path.
- #27451 — distinct branch-table-compaction/GC history loss without restart.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.