matrixorigin / matrixorigin/matrixone
disttae engine: workspaceSize/counter tracking inaccuracies and S3 GC gap
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
Audit of the disttae engine transaction layer identified four low-severity tracking and logging gaps. None affect memory correctness or data integrity — they are observability / garbage-accumulation issues.
## Findings
### D1 — `compactDeletionOnObjsLocked` does not decrement workspaceSize after freeing batches
**File**: `pkg/vm/engine/disttae/txn.go` ~line 1871
```go
txn.writes[i].bat.Clean(txn.proc.GetMPool()) // batch physically freed
txn.writes[i].bat = nil // workspaceSize NOT decremented
```
- workspaceSize retains the freed batch's original size
- Triggers false-positive BIG-TXN ( >10 MB / >100 MB ) log entries at commit time
- **Category**: monitoring false-positive (statistics only)
### D2 — `mergeTxnWorkspaceLocked` does not update workspaceSize after physical row deletion
**File**: `pkg/vm/engine/disttae/txn.go` ~line 1538
```go
txn.approximateInMemInsertCnt -= len(sels) // ✅ updated
shrinkBatchWithRowids(e.bat, sels) // batch physically shrinks
// workspaceSize unchanged // ❌ still reflects original size
```
- `deleteTableWrites` marks rows for deletion; `shrinkBatchWithRowids` removes them physically from the batch
- workspaceSize still reflects the pre-deletion batch size
- Over-estimates workspace size until the batch is flushed or the transaction ends
- **Category**: statistics over-estimation
### D3 — `RollbackLastStatement` does not reset `approximateInMemInsertSize` / `approximateInMemInsertCnt`
**File**: `pkg/vm/engine/disttae/types.go` ~line 1006
```go
for i := end; i < len(txn.writes); i++ {
txn.workspaceSize -= uint64(txn.writes[i].bat.Size()) // ✅ decremented
txn.writes[i].bat.Clean(...)
}
// approximateInMemInsertSize NOT decremented ❌
// approximateInMemInsertCnt NOT decremented ❌
```
- Rolled-back INSERT entries remain counted in the insert-approximation counters
- These counters are only reset in `dumpBatch`, never decremented on rollback
- May cause premature S3 dump triggering for the next statement in the same transaction
- **Category**: performance (premature flush)
### D4 — Async S3 object GC failures are silently logged
**File**: `pkg/vm/engine/disttae/types.go` ~line 880
```go
pool.Submit(func() {
if err := txn.engine.fs.Delete(...); err != nil {
logutil.Warnf(...) // logged but caller never sees it
}
})
```
- `pool.Submit` failure panics (correct)
- Goroutine-level `fs.Delete` failure only emits `Warnf` — callers on commit/rollback paths never learn about the failure
- Leads to silent S3 garbage object accumulation
- **Category**: S3 garbage accumulation (no data-correctness impact)
## Severity
**LOW** for all four — statistics, monitoring false-positives, premature flushes, and garbage accumulation. No memory leaks or data-correctness concerns.
## Suggested Fix
All four can be addressed in a single cleanup PR:
1. **D1**: add `txn.workspaceSize -= uint64(txn.writes[i].bat.Size())` before or alongside `bat.Clean`
2. **D2**: recompute and subtract the size delta after `shrinkBatchWithRowids`
3. **D3**: decrement `approximateInMemInsertSize`/`Cnt` for INSERT-type entries in the rollback loop
4. **D4**: propagate the `fs.Delete` error back to the caller (e.g. via an error channel or by making the caller wait for the goroutine)
Contributor guide
Assessment
This issue has not been assessed yet.