matrixorigin / matrixorigin/matrixone

bug(lineage-gc): defer expected SI write conflicts as contention

Open
#27,867 0 comments 0 reactions 1 assignee Claimed by @gouhongshen View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

`data_branch_lineage_gc` validates its fixed-SI discovery by writing the shared lineage-owner lifecycle row before commit. If an owner publisher commits after discovery, the validation transaction correctly loses with `ErrTxnWWConflict`. However, `isDataBranchLineageGCContention` does not classify `ErrTxnWWConflict` as expected contention, so `dataBranchLineageGCExecutorWithBudget` returns an error instead of safely deferring the maintenance run.

Safety is preserved because the stale GC transaction rolls back. The bug is operational/liveness behavior: normal owner-vs-GC races are reported as task failures and can cause recurring error logs, failed task metrics, and scheduler retries during snapshot/PITR/restore/DDL activity.

## Reproduction

Validated on `10.222.1.55` using NVMe storage and exact main head `d57753d442c5c19d9c490c4c393fba80f1e717f5` (which contains #27826).

A deterministic real two-CN embedded test performs:

1. CN1 starts an explicit SI transaction and reads active PITRs, fixing the GC discovery snapshot.
2. CN0 creates and commits a cluster PITR, crossing the lifecycle row in the normal frontend path.
3. CN1 writes `LineageOwnerLifecycleLockSQL()` with `WaitPolicy_FastFail` and commits.

Five consecutive runs produced the expected real storage error:

```text
update mo_catalog.mo_feature_registry ... feature_code = SNAPSHOT
...
tn.handle.commit.error error="w-w conflict: tableID: 0"
```

The returned error is `moerr.ErrTxnWWConflict` (`20619`). This is the exact stale-discovery validation race described in `compactExpiredAlterDataBranchLineageBatchWithExecutor`.

## Root cause

`pkg/sql/compile/alter_lineage_gc.go:isDataBranchLineageGCContention` includes:

- `ErrLockConflict`
- `ErrLockWaitTimeout`
- `ErrTxnNeedRetry`
- `ErrTxnNeedRetryWithDefChanged`

but omits `ErrTxnWWConflict`. The focused regression `TestDataBranchLineageGCExecutorDefersAfterContentionRollback` mirrors the same incomplete list, so mock coverage did not exercise the real SI commit result.

## Expected behavior

Treat `ErrTxnWWConflict` from this private SI GC work unit as expected contention after first giving parent cancellation precedence, exactly like the existing retry/lock conflict cases. The transaction must remain rolled back and the next scheduled invocation must rediscover from a fresh snapshot.

Add both:

- a focused executor regression using `moerr.NewTxnWWConflictNoCtx(...)`; and
- a real two-CN phase-controlled test where PITR publication lands between discovery and lifecycle validation.

## Regression source

Introduced by the GC contention classification in #27826 (`f91ce0647f5e1d8d9dcfb38a4d87063c739f074c`).

## Relationship to #27831

This condition was introduced by #27826 and remains independent of #27831's monotonic PITR-retention work. #27831 may exercise the same lineage-GC scheduler area, but it does not change this contention classification; the issue is therefore non-blocking for #27831 and must be fixed with its own paired contention regression.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.