matrixorigin / matrixorigin/matrixone

[Bug]: ISCP sync iteration is an unbounded work unit: no per-iteration events/bytes/time cap, whole-interval atomic watermark, full-interval replay

Open
#26,973 0 comments 0 reactions 1 assignee Assigned to @jiangxinmeng1 View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

### Branch Name

main

### Commit ID

c3ba9c7df14388956cb38d78681c6c35b75c42b8

### Other Environment Information

```Markdown
- Hardware parameters:
- OS type:
- Others:
```

### Actual Behavior

One ISCP sync **iteration** — the unit between watermark advances — is not bounded by
events, rows, bytes, or time. The engine collects the entire `[fromTS, toTS]` interval,
applies it, and advances the watermark **once, atomically, at the end**, inside a single
transaction. A long catch-up (e.g. a 30-minute gap at 100 updates/s ≈ 360k change rows)
is therefore a single unbounded work unit: unbounded transaction size, unbounded retry
replay, and — for index consumers that persist a chronological frame per op-run — an
unbounded persistent frame/segment count and query-time load cost. A timeout before the
watermark advances replays the same large interval.

This is an **ISCP-engine availability property shared by all index consumers** (hnsw,
ivfflat, cagra, ivfpq, bm25, fulltext, fulltext2), not specific to any one index.


- Interval is collected whole, not chunked: `pkg/iscp/iteration.go:240`
`CollectChanges(ctxWithoutTimeout, rel, iterCtx.fromTS, iterCtx.toTS, mp)` — no
time/row/event/byte split of the interval.
- The per-blob `Full()` cap (`MAX_CDC_DATA_SIZE = 8192`, `pkg/iscp/index_sqlwriter.go:36`)
and per-statement tuple batching bound one flush's round-trip, **not** the iteration.
- Watermark advances once, at channel close, inside the same txn as all persisted work:
- `pkg/iscp/index_consumer.go:242` and `:377` (hnsw/ivfflat/bm25/fulltext)
- `pkg/iscp/cuvs_writer.go:394` (cagra/ivfpq)
- `pkg/iscp/fulltext2_consumer.go:115` (fulltext2)
- Replay unit on failure is the whole interval: on error the status keeps `From`, so the
next iteration re-reads `[fromTS, toTS]` from the last checkpoint
(`pkg/iscp/iteration.go:335`).

## Impact

- **Transaction size**: all changes for the interval land in one txn. A large catch-up can
exceed txn/memory limits.
- **Retry/timeout**: any failure before the atomic watermark advance replays the entire
interval — the interval never shrinks, so a persistently large gap can livelock.
- **Persistent frame/segment count** (frame-per-op-run consumers such as fulltext2/bm25):
N op-runs → N persisted frames until a separate MERGE/idxcron compaction runs; nothing in
the CDC path collapses them (`pkg/fulltext2/storage.go` `LoadTailSegments` reconstructs one
in-memory `Segment` per insert frame; the byte-budget guard bounds bytes, not count).
- **Query cost**: every query walks the full accumulated frame/segment set until compaction.

### Expected Behavior

## Proposed fix (either is acceptable)

1. **Bound the iteration**: cap each ISCP iteration by events/bytes/work and advance an
**idempotent** watermark per bounded unit, so a large gap is drained in many small,
independently-committed steps and a retry replays only the last bounded unit. This is an
engine-level change in `pkg/iscp` and benefits every consumer.
2. **Bounded incremental commit contract**: persist/coalesce the chronological work
incrementally with an equivalent bounded-commit contract that advances the watermark per
bounded unit while preserving last-writer-wins ordering.

### Steps to Reproduce

```Markdown
Create IVFFLAT index async index and insert data via CDC.
```

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.