matrixorigin / matrixorigin/matrixone

disttae engine: workspaceSize/counter tracking inaccuracies and S3 GC gap

Open
#25,204 2 comments 0 reactions 1 assignee Claimed by @gouhongshen View on GitHub
ai-medium deferred kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

Audit of the disttae engine transaction layer identified four low-severity tracking and logging gaps. None affect memory correctness or data integrity — they are observability / garbage-accumulation issues.

## Findings

### D1 — `compactDeletionOnObjsLocked` does not decrement workspaceSize after freeing batches

**File**: `pkg/vm/engine/disttae/txn.go` ~line 1871

```go
txn.writes[i].bat.Clean(txn.proc.GetMPool()) // batch physically freed
txn.writes[i].bat = nil // workspaceSize NOT decremented
```

- workspaceSize retains the freed batch's original size
- Triggers false-positive BIG-TXN ( >10 MB / >100 MB ) log entries at commit time
- **Category**: monitoring false-positive (statistics only)

### D2 — `mergeTxnWorkspaceLocked` does not update workspaceSize after physical row deletion

**File**: `pkg/vm/engine/disttae/txn.go` ~line 1538

```go
txn.approximateInMemInsertCnt -= len(sels) // ✅ updated
shrinkBatchWithRowids(e.bat, sels) // batch physically shrinks
// workspaceSize unchanged // ❌ still reflects original size
```

- `deleteTableWrites` marks rows for deletion; `shrinkBatchWithRowids` removes them physically from the batch
- workspaceSize still reflects the pre-deletion batch size
- Over-estimates workspace size until the batch is flushed or the transaction ends
- **Category**: statistics over-estimation

### D3 — `RollbackLastStatement` does not reset `approximateInMemInsertSize` / `approximateInMemInsertCnt`

**File**: `pkg/vm/engine/disttae/types.go` ~line 1006

```go
for i := end; i < len(txn.writes); i++ {
txn.workspaceSize -= uint64(txn.writes[i].bat.Size()) // ✅ decremented
txn.writes[i].bat.Clean(...)
}
// approximateInMemInsertSize NOT decremented ❌
// approximateInMemInsertCnt NOT decremented ❌
```

- Rolled-back INSERT entries remain counted in the insert-approximation counters
- These counters are only reset in `dumpBatch`, never decremented on rollback
- May cause premature S3 dump triggering for the next statement in the same transaction
- **Category**: performance (premature flush)

### D4 — Async S3 object GC failures are silently logged

**File**: `pkg/vm/engine/disttae/types.go` ~line 880

```go
pool.Submit(func() {
if err := txn.engine.fs.Delete(...); err != nil {
logutil.Warnf(...) // logged but caller never sees it
}
})
```

- `pool.Submit` failure panics (correct)
- Goroutine-level `fs.Delete` failure only emits `Warnf` — callers on commit/rollback paths never learn about the failure
- Leads to silent S3 garbage object accumulation
- **Category**: S3 garbage accumulation (no data-correctness impact)

## Severity

**LOW** for all four — statistics, monitoring false-positives, premature flushes, and garbage accumulation. No memory leaks or data-correctness concerns.

## Suggested Fix

All four can be addressed in a single cleanup PR:

1. **D1**: add `txn.workspaceSize -= uint64(txn.writes[i].bat.Size())` before or alongside `bat.Clean`
2. **D2**: recompute and subtract the size delta after `shrinkBatchWithRowids`
3. **D3**: decrement `approximateInMemInsertSize`/`Cnt` for INSERT-type entries in the rollback loop
4. **D4**: propagate the `fs.Delete` error back to the caller (e.g. via an error channel or by making the caller wait for the goroutine)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.