[Feature] Track incremental computation with row binlog, Table Stream, and MTMV

### Description

This issue tracks the end-to-end incremental computation capability in Apache Doris.

The implementation is being built in three layers:

```text
OLAP table DML
-> row binlog: durable row-level changes, before images, operation type, LSN/TSO
-> Table Stream: change-query semantics, per-partition consumption offsets, snapshots and reset
-> MTMV incremental maintenance: derive and transactionally apply only the affected result changes
```

This tracker replaces the stale umbrella issue #57921 and incorporates the row-binlog foundation originally tracked by #61956.

### Goals

- Provide a durable row-level change log for Doris internal tables.
- Expose deterministic incremental consumption through Table Stream.
- Provide snapshot reads aligned with stream consumption points for joins and recovery.
- Integrate Table Stream with MTMV planning and refresh execution for row-level incremental maintenance.
- Preserve correctness across replicas, compaction, schema changes, concurrent consumers, failures, retries, and restart/replay.
- Add complete unit, regression, compatibility, and failure-recovery coverage.
- Document supported table models, SQL syntax, semantics, limitations, and operational guidance.

### Scope boundary

This tracker covers the internal row-binlog and Table Stream stack used by incremental computation, plus its MTMV integration. External CDC Streaming Jobs and general MTMV partition refresh or query-rewrite changes are out of scope unless they directly modify this stack.

FE-side (CCR) binlog is listed under "Adjacent work" below: it is a separate feature, but several correctness issues tracked here live in that code and are reported against the same stack.

The repository review has been refreshed through **2026-09-01**. PR #61382 predates the original review window but is included because it is the Table Stream metadata and DDL foundation.

### Status summary (2026-09-01)

| Layer | Merged | Open / Draft | Assessment |
| --- | --- | --- | --- |
| Row binlog | 23 | 7 | Main path complete; remaining work is scheduling locality, partial update, and throughput. |
| Table Stream | 17 | 6 | Local and cloud paths complete; metadata lifecycle and DDL surface still have gaps. |
| MTMV incremental maintenance | 0 | 1 draft | Implemented on a branch, but nothing has been merged. **This is the critical path.** |

### Progress

#### Row binlog foundation

Merged — core path:

- [x] #62058 — define row-binlog metadata and schema.
- [x] #63110 — implement row-binlog read and write paths.
- [x] #63643 — implement replica clone, scheduling, and compaction.
- [x] #64133 — allocate LSNs at the sink and keep LSN/TSO semantics consistent across replicas.
- [x] #65810 — support cloud binlog and an independent binlog tablet.
- [x] #66883 — derive row binlog in the segment-writer transform chain and delete `RowBinlogSegmentWriter`. Supersedes #64674.
- [x] #66889 — add a per-row LSN column for detail (DUP) tables, required by MTMV incremental maintenance.

Merged — fixes:

- [x] #64249, #65032 — cumulative policy NPE during binlog scheduling and binlog compaction.
- [x] #64545, #65663, #65686 — incomplete binlog metadata in the grouped rowset writer and its tests.
- [x] #65076 — include hidden key columns while excluding hidden non-key internal columns.
- [x] #65438 — align the row-binlog AFTER schema with before values.
- [x] #65481 — report missing row binlog for INCR queries.
- [x] #66046 — emit INSERT rather than UPDATE when an insert follows a delete.
- [x] #66093 — disallow Row Binlog on MOW tables with cluster keys.
- [x] #66432 — fix the BE crash when an APPEND_ONLY row-binlog scan omits key columns.
- [x] #66553 — wrap Row Binlog AFTER columns as nullable before conversion.
- [x] #66601 — support complex `array_agg` state serialization.
- [x] #66648 — avoid mutating shared source columns during concurrent Row Binlog segment flushes.
- [x] #66665 — fix row-binlog routing for auto partitions.
- [x] #66840 — allow key-only partial updates with row binlog.

Open:

- [ ] #66191 — keep row binlog tablets colocated with base tablets. Approved by five reviewers but not merged; #67092 and #67106 are correctness issues on this path.
- [ ] #66899 vs #66764 — flexible partial update. #66899 adds support, #66764 rejects it at DDL time. **These are mutually exclusive and a decision is needed** before #65866 can be closed.
- [ ] #67050 — resolve before columns by unique ID.
- [ ] #66963 — restrict state types to aggregate tables (HLL/QuantileState path behind #65865).
- [ ] #64484, #64495 — missing binlog column index, and timestamp nullable mismatch. Open since mid-June; need confirmation of whether #66883 already covers them.
- [ ] #65947 — enabling binlog reduces import throughput by about 65%. The only attempted fix (#66894) was closed; **no owner and no current fix**. This blocks enabling the feature by default.

#### Table Stream foundation

Merged — core path:

- [x] #61382 — add stream metadata, DDL, and information-schema visibility.
- [x] #62453 — add stream query and transactional consumption infrastructure.
- [x] #63850 — connect OLAP Table Stream to row binlog and add DETAIL, APPEND_ONLY, and MIN_DELTA scan modes.
- [x] #64776 — add stream snapshot/reset reads, table incremental reads, and time-travel snapshot support.
- [x] #65859 — add the Cloud Table Stream control plane.
- [x] #66743 — add comprehensive Cloud Table Stream coverage.

Merged — fixes:

- [x] #65468, #65657 — stream scan normalization, snapshot/reset pruning, and partition-prune state propagation.
- [x] #65636 — rename the misleading `StreamTableInfo`.
- [x] #65826 — MIN_DELTA delete op returns the first op's before image.
- [x] #65973 — force binlog TSO predicate pushdown and forbid other pushdowns.
- [x] #66338 — fix multi-segment row-binlog reads for MoW tables.
- [x] #66683 — parse stream properties before base-table validation so valid APPEND_ONLY and DETAIL streams are not rejected.
- [x] #66739, #67079 — localize Cloud Table Stream rewrite state handling and its test hook.
- [x] #67126 — commit table stream offsets for empty inserts.
- [x] #67173 — remove the base-table cache and use a dynamic schema.

Open:

- [ ] #65274 — expand Table Stream negative and exception-path regression coverage. Draft since 2026-07-06; without it the negative paths have no regression protection.
- [ ] #66287 and #65698 — both mark a stream stale/unavailable after its base table is dropped (#65389). **Duplicate effort; one needs to be picked.** Note: #66287 was previously checked off in #65265 but is still a draft.
- [ ] #65959 — fix table stream change-type predicate pushdown.
- [ ] #67130 — derive table stream schema from the base table (#67109).
- [ ] #67181 — wait for running transactions before incremental reads in cloud mode.
- [ ] #65388, #65390 — `ALTER STREAM SET COMMENT` and `CANCEL ALTER STREAM` are not supported. No owner.

#### MTMV incremental maintenance

**Corrected.** The previous revision of this tracker stated that no MTMV implementation PR had been identified. That is no longer accurate:

- #66719 is the focused tracker for this layer.
- #62606 is the implementation PR. It has been open since 2026-04-20 and is **still a draft: +47922 / -889 across 343 files and 273 commits, with no review and no approval.** #66266 and #61496 are pipeline-test drafts from the same branch.

Per #66719, the branch already covers projection, filter, aggregate (COUNT/SUM/AVG/MIN/MAX, bitmap aggregates, expression arguments, bare GROUP BY), INNER/CROSS/LEFT OUTER/FULL OUTER joins, UNION ALL, subquery aliases, `REFRESH MATERIALIZED VIEW ... INCREMENTAL/PARTITIONS`, dry-run, automatic full-refresh fallback, chained IVM MTMV, `ivm_use_full_keys`, and regression tests under `mtmv_p0/ivm`.

The blocker is not implementation, it is **reviewability and landing**. A single PR of this size cannot be reviewed effectively, and this layer currently has no assigned reviewer.

- [ ] Split #62606 into independently mergeable PRs (delta plan derivation, per-operator incremental semantics, MTMV lifecycle integration, atomic result-apply plus offset advancement, fallback and observability, tests).
- [ ] Assign FE and storage reviewers for the split PRs.
- [ ] Define supported MTMV query shapes and eligibility/fallback rules.
- [ ] Derive delta plans from Table Streams, including snapshot sides for joins.
- [ ] Define incremental semantics for projection, filter, join, union, aggregation, and supported expressions.
- [ ] Integrate stream creation and lifecycle with MTMV creation, alteration, pause/resume, and drop.
- [ ] Execute result changes and stream-offset advancement atomically.
- [ ] Support full-refresh bootstrap/reset and fallback when incremental maintenance is not valid.
- [ ] Define retry, idempotency, concurrent refresh, failover, and EditLog replay behavior.
- [ ] Add observability for refresh mode, consumed offsets, lag, fallback reason, and failures.
- [ ] Add FE and BE unit tests plus end-to-end regression tests for correctness and recovery.
- [ ] Add user and operator documentation.

#### Adjacent work

- #61198 — global monotonically increasing timestamp service (TSO). Closed.
- #65978 / #66191 — row binlog tablet locality during scheduling.
- #66930 — preserve binlog config across REPLACE TABLE (#66571, #67105).
- #65484 — cover binlog config validation routing (#65383).
- #67156 — single replica ingest binlog.
- #63134 — snapshot creation fails with a hard-link NOT_FOUND error when binlog is enabled during a CLONE task.

### Bug and correctness backlog

#65265 is the consolidated bug and test backlog. Current state:

- Bug list: 7 of 17 closed.
- Correctness issues: 3 of 22 closed (#66273, #66338, #67095). #66287 is checked off there but is still a draft PR.
- **17 correctness issues have no linked PR and no assignee.**

The unowned correctness issues (#67091–#67094, #67096–#67108) were filed together on 2026-08-25 and are design-level defects rather than isolated bugs: BE restart recovery, replica repair and balancing, TSO boundary semantics, cloud publish/commit failure handling, leader/follower metadata divergence, and lock/scan target disagreement after RENAME or REPLACE TABLE. They need owners before they can be scheduled.

### Correctness and compatibility checklist

- [ ] DUP and supported UNIQUE KEY MOW table semantics are documented and tested.
- [ ] Full update, partial update, delete, sequence column, hidden key, and schema-change paths are covered.
- [ ] LSN/TSO ordering is deterministic across replicas and remains valid after clone and compaction.
- [ ] Concurrent stream consumption either commits data and offsets together or fails without advancing offsets.
- [ ] Snapshot, reset, and incremental reads agree at partition boundaries and after restart.
- [ ] Binlog retention cannot silently invalidate an active stream or MTMV; stale state is observable and recoverable.
- [ ] Unsupported plans fail clearly or use a documented full-refresh fallback.
- [ ] Cloud-mode support or exclusion is explicit.
- [ ] Upgrade, downgrade, rolling restart, backup/restore, and replication behavior is defined before general availability.
- [ ] Enabling binlog does not regress import throughput to an unacceptable degree (#65947).

### Overlapping or superseded PRs

- #64137 — closed; its compaction-policy crash path is covered by #64249.
- #64484 and #64495 — overlap with the metadata and nullable-column fixes merged in #64545; needs confirmation against #66883.
- #64674 — closed; superseded by #66883.
- #66664 — closed; the HLL/state-type path is now handled by #66963.
- #66894 — closed; #65947 remains unresolved.

### Exit criteria

- Row binlog and Table Stream semantics are stable and documented.
- At least one supported MTMV query class refreshes incrementally end to end with atomic offset advancement.
- Correctness, failure recovery, replay, compaction, and schema evolution tests pass in the supported deployment modes.
- Unsupported cases have explicit diagnostics and a documented fallback.
- Remaining limitations and follow-up work are tracked by linked issues or PRs.