[STIP-30][Feature][CDC] : Unified CDC lag and progress observability model

### Search before asking

- [x] I searched existing feature and CDC issues and found no equivalent proposal.

### Description

This is STIP-30, a design proposal for #11354.

SeaTunnel CDC connectors expose different source positions and lifecycle state: MySQL binlog files and GTIDs, PostgreSQL LSNs, Oracle SCNs, TiDB timestamps, snapshot splits, and checkpointed reader state. Operators currently need connector logs or checkpoint internals to determine whether a job is snapshotting, catching up, consuming incrementally, recovering, idle, or no longer progressing.

The proposal defines a normalized latest-progress view while preserving connector-native positions and without claiming that every connector supports the same lag semantics.

## Goals

- expose current snapshot, incremental, checkpoint, and restore progress
- keep connector-native position details available for diagnosis
- distinguish CDC lifecycle, recovery state, and runtime health
- represent unsupported or approximate values explicitly
- keep v1 additive and latest-snapshot only
- validate the model through complete MySQL and PostgreSQL runtime paths before stabilizing the public contract

## Non-goals for v1

- no historical timeline or retention store
- no UI dependency
- no universal numeric lag across connectors
- no inference that an unchanged position means backpressure or a stalled source
- no unbounded list of completed split details in job detail
- no change to CDC checkpoint or restore behavior

## Design principles

1. Connectors report only facts they own.
2. The engine adds job, pipeline, vertex, task, observation, and aggregation context.
3. Snapshot assignment and reader consumption remain separate report sources.
4. `EXACT` is used only when current runtime state proves a value.
5. Missing, stale, unsupported, and best-effort values are not replaced with zero or guessed values.
6. The implementation is validated internally before the public `seatunnel-api` contract is stabilized.

## Contract layers

| Layer | Responsibility |
| --- | --- |
| CDC reader report | Current consumed position, active split, source-event time, checkpoint candidate/completion state, and restore state owned by that reader |
| CDC enumerator report | Snapshot assignment, completed split watermarks, prepared remaining splits, remaining unchunked tables, and source-level lifecycle information |
| Engine snapshot | Runtime identity, report observation time, stale/missing report state, task aggregation, and latest-snapshot storage |
| REST response | Additive `cdcProgress` section built from the latest engine snapshot |

During validation, report and snapshot types remain experimental. The final public types are stabilized only after MySQL and PostgreSQL prove the same contract across two offset families.

## Reporting and transport path

The connector-facing provider is a local, non-blocking capability. It returns an immutable snapshot already maintained by the reader or enumerator. It does not perform remote calls and is not invoked directly from REST.

The preferred v1 transport uses separate collection paths for reader and enumerator state:

- a worker-side engine sampler runs from `TaskExecutionService` on the configured reporting interval
- it asks source reader tasks that implement the experimental progress capability for their latest immutable report
- worker reader reports are batched and sent to the coordinator through one engine operation
- the coordinator discovers enumerator placement from the current job plan and slot assignments, then requests the immutable report from the hosting member; the enumerator is not necessarily master-local and is not sampled through the reader task loop
- report collection does not run inside `pollNext()`, record emission, enumerator `run()`, or the checkpoint lock
- the coordinator aggregates reader and enumerator reports by source vertex
- the coordinator rejects older reader reports using source vertex, task index, execution attempt, and report sequence
- the latest state store overwrites reports by reader/enumerator identity instead of appending history
- REST reads the coordinator-side latest snapshot

Lifecycle callbacks update local immutable report state immediately. Transport remains periodic and coalesced, so every record does not create a worker/master operation. A later optimization may request an immediate asynchronous flush for phase or restore transitions, but correctness must not depend on that flush.

```mermaid
sequenceDiagram
participant DB as MySQL source
participant Reader as CDC reader on worker
participant Sampler as Worker reader sampler
participant Coordinator as Coordinator progress collector
participant Enumerator as CDC enumerator on its hosting member
participant Store as Latest progress state store
participant REST as Job detail REST

DB->>Reader: record or heartbeat
Reader->>Reader: update immutable reader report
Enumerator->>Enumerator: rebuild assignment and watermark report from current state

Note over Reader,Enumerator: pollNext and checkpoint callbacks never perform REST or state-store writes

Sampler->>Reader: read latest reader report
Sampler->>Coordinator: batch reader reports by task and attempt identity
Coordinator->>Enumerator: request latest enumerator report from its hosting member
Coordinator->>Coordinator: reject older attempts/sequences and aggregate by source vertex
Coordinator->>Store: overwrite latest reader/enumerator/source snapshots
REST->>Store: read latest CDC progress
Store-->>REST: source summary and bounded active task details
```

## Reporting triggers and consistency

Local report state may request an asynchronous refresh on:

- snapshot split assignment or completion
- transition between snapshot, catch-up, and incremental lifecycle stages
- emitted data or heartbeat position change
- `snapshotState(checkpointId)` candidate creation
- `notifyCheckpointComplete(checkpointId)` promotion
- `notifyCheckpointAborted(checkpointId)` candidate removal
- explicit restore notification

These events are refresh signals only. They are not the source of truth for assigned, completed, running, or checkpoint counts because events can be duplicated, delayed, or lost during failure and restore. Exact values are rebuilt from the latest reader and enumerator state.

Periodic transport coalesces multiple local changes. Every reader report contains an execution attempt identity, a monotonically increasing sequence within that attempt, and timestamps. The coordinator keeps the newest report for `(sourceVertexId, taskIndex, executionAttemptId)` and rejects reports from superseded attempts. Reports from different tasks are not treated as one atomic distributed snapshot; the response exposes freshness and partial aggregation so temporary inconsistency is visible.

## Report scope and aggregation

### Enumerator-level report

- assigned split count
- completed split count
- running split count
- prepared but unassigned split count
- remaining tables not yet chunked
- completed low/high watermarks keyed by `splitId`
- enumerator lifecycle stage

### Reader-task report

- task index, execution attempt identity, and report sequence
- active split and table when applicable
- current consumed position
- last completed checkpoint position
- restored position and recovery state
- relevant timestamps

### Source/vertex summary

- task count, fresh report count, stale report count, and missing report count
- lifecycle stage only when it can be derived without hiding mixed task states
- aggregate snapshot counts with field-level accuracy
- health summary separate from lifecycle and recovery

For parallel readers, the response does not invent one top-level `splitId`, `tablePath`, or current position. Active task details remain separate. If reports disagree, the summary reports a mixed or unknown lifecycle state.

## Snapshot progress and watermarks

Snapshot details are keyed by `splitId`. A split and watermark are never selected independently from unordered maps.

The model distinguishes:

- assigned splits
- completed splits
- running splits
- prepared remaining splits
- remaining tables not yet chunked

With lazy chunking, assigned and completed counts can be exact while final total split count and overall percentage remain unknown. Accuracy is therefore represented per field rather than only once for the whole snapshot group.

Low and high watermarks are optional connector-native values. They are reported only when associated with the same split and are not required for every CDC connector.

## Incremental and checkpoint progress

```text
record or heartbeat emitted
-> update currentConsumedPosition and lastPositionChangeAt

snapshotState(checkpointId)
-> retain checkpoint candidate for checkpointId

notifyCheckpointComplete(checkpointId)
-> promote candidate to lastCompletedCheckpointPosition

notifyCheckpointAborted(checkpointId)
-> discard candidate without changing the last completed checkpoint
```

Startup position, current consumed position, checkpoint candidate, last completed checkpoint position, and restored position are separate facts. One value is not reused for another unless the runtime lifecycle proves they are equal.

Until candidate retention and completion/abort callbacks are wired through the real reader lifecycle, `lastCompletedCheckpointPosition` is reported as `UNSUPPORTED`. Neither startup position nor current consumed position proves checkpoint completion.

## Restore origin

`SourceReader.addSplits(...)` is used for both normal assignment and restore, so restore must not be inferred from that method alone.

The preferred design is an additive engine-provided restore callback invoked before restored splits are installed:

```java
default void notifyRestored(SourceRestoreContext context) throws Exception {}
```

The restore context contains the restored checkpoint identity when available, the execution attempt or restore epoch, and the restored split identities/positions. An alternative context accessor can be considered during API review, but the accepted design must provide explicit engine origin rather than connector inference.

The reader records:

- restored checkpoint identity
- restored position
- first position observed after restore
- recovery transition state

## Lifecycle, recovery, and health

These are separate dimensions.

### CDC lifecycle

- `SNAPSHOT`
- `CATCH_UP`
- `INCREMENTAL`
- `UNKNOWN`

For MySQL, `CATCH_UP -> INCREMENTAL` occurs when `IncrementalSplitState.markEnterPureIncrementPhaseIfNeed(position)` confirms that the consumed position has reached or passed `maxSnapshotSplitsHighWatermark`. The existing `CompletedSnapshotPhaseEvent` communicates that transition to the enumerator.

### Recovery state

- `RESTORING`
- `RESTORED`
- `STEADY`
- `UNKNOWN`

### Health state

- `HEALTHY`
- `STALE_REPORT`
- `NO_PROGRESS_OBSERVED`
- `CONNECTOR_REPORTED_STALLED`
- `UNKNOWN`

Backpressure remains a separate runtime metric. The CDC model does not infer backpressure from an unchanged position.

## Time semantics

| Field | Meaning |
| --- | --- |
| `observedAt` | Engine time when the report was collected |
| `lastPositionChangeAt` | Last time the reader observed a different consumed position |
| `lastSourceEventAt` | Source event time when the connector can provide it |
| `sourceHighWatermarkObservedAt` | Time when the reported source head/high watermark was observed |

An unchanged offset may represent an idle source, downstream backpressure, checkpoint coordination, stale reporting, or actual lack of progress. Without evidence that the source head is ahead of the consumed position, the framework reports `NO_PROGRESS_OBSERVED` or `STALE_REPORT`, not `CONNECTOR_REPORTED_STALLED`.

## STIP status, validation scope, and stabilization gate

#11354 remains the feature umbrella and this STIP remains the design source of truth.

MySQL and PostgreSQL are the validation scope for v1, not a release promise. No target release is proposed until the stabilization evidence below is complete and the community agrees that the contract is ready.

The implementation and exposure remain phased:

- the first slice contains internal worker/coordinator transport and experimental connector-facing reports only
- it does not expose REST fields, metrics, or a stable `schemaVersion = 1` contract
- after the stabilization gate, the first consumption surface is an optional additive `cdcProgress` section in job detail
- v1 does not add a dedicated progress endpoint
- bounded metrics follow only after freshness and lifecycle semantics are validated; connector-native positions never become metric labels

The experimental reports can become stable public `seatunnel-api` DTOs and freeze `schemaVersion = 1` only after:

1. MySQL binlog positions and PostgreSQL LSNs validate the same ownership, nullability, and cardinality rules end to end.
2. Worker reader collection, coordinator-owned enumerator discovery/collection, attempt/sequence ordering, stale-report rejection, serialization, and lifecycle cleanup are covered by tests.
3. Checkpoint candidate, completion, and abort handling and explicit engine-provided restore identity are implemented and validated through recovery.
4. Multi-source jobs are verified without merging connector types, positions, or task identities.
5. Unsupported, unavailable, best-effort, and stale data are represented consistently without guessed defaults.
6. The REST shape, compatibility impact, and connector limitations receive documentation and community review.

Value quality and report freshness remain separate dimensions:

| State | Meaning |
| --- | --- |
| `EXACT` | The current connector/runtime state proves the value without approximation. |
| `BEST_EFFORT` | A value is available and useful, but its precision is not guaranteed. |
| `UNSUPPORTED` | The connector or current implementation cannot provide the field. |
| `UNAVAILABLE` | The field is supported but no value is available for this observation. |
| `STALE` | The report exceeded the freshness threshold. This is report health, not field accuracy; existing value quality is retained while the report is marked stale. |

## API exposure and compatibility

- REST exposure is an additive optional `cdcProgress` field on the existing job-detail response; existing fields and endpoints do not change
- `cdcProgress` is absent for jobs without supported CDC sources and may be absent while no report has been accepted yet
- the response carries `schemaVersion` and contains one entry per CDC source vertex
- connector-native positions are exposed only in the bounded REST detail response
- CDC metrics are registered through the existing job metrics path without renaming or changing existing metrics
- the connector-facing SPI and internal report types remain experimental during MySQL and PostgreSQL validation
- stable public `seatunnel-api` DTOs are introduced only after both runtime paths validate ownership, nullability, cardinality, and restore/checkpoint semantics

## Canonical v1 response shape

The response below is the target contract to validate. Exact Java class names may change during implementation, but ownership, cardinality, nullability, and semantics should not.

```json
{
"cdcProgress": {
"schemaVersion": 1,
"observedAt": 1784023200000,
"sources": [
{
"pipelineId": 1,
"vertexId": "source-1",
"connectorType": "MySQL-CDC",
"lifecycle": "CATCH_UP",
"recovery": {
"state": "STEADY",
"restoredCheckpointId": null
},
"health": {
"state": "HEALTHY",
"source": "FRAMEWORK",
"reason": null
},
"summary": {
"readerTaskCount": 4,
"freshReportCount": 4,
"staleReportCount": 0,
"missingReportCount": 0,
"assignedSplitCount": 12,
"completedSplitCount": 8,
"runningSplitCount": 4,
"preparedRemainingSplitCount": 3,
"remainingUnchunkedTableCount": 2
},
"tasks": [
{
"taskIndex": 0,
"executionAttemptId": "attempt-2",
"reportSequence": 41,
"observedAt": 1784023200000,
"lastPositionChangeAt": 1784023199000,
"lastSourceEventAt": 1784023198500,
"lifecycle": "CATCH_UP",
"activeSplit": {
"splitId": "inventory.products:3",
"tablePath": "inventory.products"
},
"currentConsumedPosition": {
"type": "MYSQL_BINLOG",
"schemaVersion": 1,
"values": {
"file": "mysql-bin.000042",
"position": "1842"
}
},
"lastCompletedCheckpointPosition": null,
"restoredPosition": null
}
],
"accuracy": {
"summary.assignedSplitCount": "EXACT",
"summary.completedSplitCount": "EXACT",
"summary.finalTotalSplitCount": "UNSUPPORTED"
},
"detailsTruncated": false
}
]
}
}
```

Nullability and invariants:

- unsupported or unavailable values are absent or `null`, not zero-filled
- position objects always include position type and schema version
- accuracy has one canonical source and can differ by field
- recovery and health do not replace lifecycle
- stale or missing task reports remain visible in summary counts
- a completed checkpoint position exists only after completion notification
- source entries are keyed by pipeline and vertex identity; jobs with multiple CDC sources do not merge connector types or positions

## Payload boundary

The job-detail `cdcProgress` response contains:

- one summary per CDC source vertex
- at most one latest detail per active reader task
- active split detail only
- aggregate counts for completed, prepared, and remaining work

It does not include historical reports or every completed split. Task details are bounded by configured source parallelism. If a deployment applies a response-entry limit, the response sets `detailsTruncated=true`; a paginated detail endpoint can be added later without changing the summary contract.

## Metrics boundary

REST keeps connector-native diagnostic values. GTIDs, LSNs, SCNs, resolved timestamps, split IDs, table paths, and raw position maps are never exported as metric labels.

The proposed v1 metrics expose only bounded runtime facts:

| Metric | Value | Allowed dimensions |
| --- | --- | --- |
| `seatunnel_cdc_progress_report_age_seconds` | Age of the latest accepted report | job, pipeline, source vertex, task index, connector type |
| `seatunnel_cdc_progress_missing_reader_reports` | Number of expected reader reports that are missing | job, pipeline, source vertex, connector type |
| `seatunnel_cdc_progress_stale_reader_reports` | Number of stale reader reports | job, pipeline, source vertex, connector type |
| `seatunnel_cdc_progress_no_progress_duration_seconds` | Time since the consumed position last changed | job, pipeline, source vertex, task index, connector type |
| `seatunnel_cdc_progress_lifecycle` | One-hot lifecycle state | job, pipeline, source vertex, connector type, lifecycle |
| `seatunnel_cdc_progress_recovery_state` | One-hot recovery state | job, pipeline, source vertex, connector type, recovery state |

The final exporter prefix and existing job/vertex identity labels will follow SeaTunnel's current metrics conventions. No metric attempts to serialize or numerically compare connector-native positions.

## Connector capability matrix

`REQUIRED` means the adapter must provide the fact before that connector is declared supported by this feature. `OPTIONAL` means it is emitted only when the runtime can prove it. `UNSUPPORTED_V1` is explicit and is not represented as zero.

| Fact | MySQL CDC | PostgreSQL CDC | Oracle CDC | TiDB CDC |
| --- | --- | --- | --- | --- |
| Native position family | Binlog file/position and optional GTID | LSN | SCN | Commit/resolved timestamp |
| Snapshot split progress | REQUIRED | REQUIRED | REQUIRED | REQUIRED |
| Current consumed position | REQUIRED | REQUIRED | REQUIRED | REQUIRED |
| Completed checkpoint position | REQUIRED after checkpoint lifecycle wiring | REQUIRED after checkpoint lifecycle wiring | REQUIRED after checkpoint lifecycle wiring | REQUIRED after checkpoint lifecycle wiring |
| Restored position and restore identity | REQUIRED after engine restore context wiring | REQUIRED after engine restore context wiring | REQUIRED after engine restore context wiring | REQUIRED after engine restore context wiring |
| Snapshot low/high watermarks | OPTIONAL per split | OPTIONAL per split | OPTIONAL per split | OPTIONAL per split |
| Source event time | OPTIONAL | OPTIONAL | OPTIONAL | OPTIONAL |
| Independently observed source-head position | OPTIONAL | OPTIONAL | OPTIONAL | OPTIONAL |
| Exact offset lag | UNSUPPORTED_V1 | UNSUPPORTED_V1 | UNSUPPORTED_V1 | UNSUPPORTED_V1 |
| Time-based event delay | OPTIONAL and not named source lag | OPTIONAL and not named source lag | OPTIONAL and not named source lag | OPTIONAL and not named source lag |

Position families remain connector-native. Time-based event delay is reported separately and is not called source lag unless a connector can also observe a comparable source-head position.

## Latest snapshot storage

- workers keep immutable local reports only until the next collection
- the coordinator stores the latest report by task identity through an engine state-store abstraction
- Hazelcast `IMap` or another backend remains an implementation detail
- ordering uses `(sourceVertexId, taskIndex, executionAttemptId, reportSequence)` so reports from an older execution attempt cannot suppress a restarted task whose sequence begins at zero
- task reports are cleaned with pipeline/job lifecycle
- checkpoint state remains the source of restore correctness; observability history is not added to checkpoints

## Validation path

1. Multi-table MySQL snapshot with parallel readers and lazy chunk creation.
2. Split/watermark association across unordered production state.
3. `CATCH_UP -> INCREMENTAL` using the real MySQL high-watermark transition.
4. Current position advancing past checkpoint N.
5. Checkpoint N completion, checkpoint N+1 abort, and preservation of the last completed checkpoint.
6. Explicit restore context, restored position, and first post-restore position.
7. Idle source, stale report, no-progress timeout, and backpressure without false stalled attribution.
8. Separate worker reader and coordinator enumerator collection paths.
9. Sequence rejection within one attempt, acceptance of sequence zero in a new attempt, and rejection of late reports from an older attempt.
10. Aggregation and REST serialization for multiple CDC source vertices using different connector types.
11. Metrics export without connector-native positions, split IDs, or table paths as labels.
12. PostgreSQL LSN implementation without changing the response contract.
13. Serialization round trips and invalid lifecycle/recovery/health combinations.

## Proposed STIP phases and task tracking

Updated 6 September 2026 (UTC). Overall status: **Under review**. The discussion supports a narrow experimental implementation; the remaining phases and public-contract stabilization still need community agreement. No target release is promised.

Feature umbrella: #11354. This issue (#11364) remains the canonical design and task tracker. No separate phase-task issues have been opened. Unchecked items below are proposed work, not claimed ownership or completed delivery. Historical PR #11395 was closed unmerged and was superseded by #11512.

Discussion: [existing dev-list thread](https://lists.apache.org/thread/nwjgvxh972mv1b1mykojmlv4olj88nnz). The current contract supersedes the opening email's group-level support and timeout-based stalled wording: value quality is per field, freshness is separate, and unchanged offsets alone do not prove a stall.

### Phase 1: Internal reporting foundation

Tracking: #11354 / #11364. Implementation: #11512, open for review. Keep its scope unchanged.

- [x] Experimental immutable reader/enumerator reports, per-field accuracy, periodic collection and latest-only storage are implemented in #11512.
- [x] Coordinator-owned enumerator discovery/collection, member-to-coordinator reader batches, attempt-local ordering, bounded active-split details and pipeline cleanup are implemented in #11512.
- [x] Consumed, completed-checkpoint and restored positions remain distinct; unwired lifecycle facts are explicitly unsupported.
- [ ] Complete exact-head CI and maintainer review, then merge the internal-only foundation.

Checked items mean code exists in the open PR, not that the STIP is implemented or the public contract accepted. No REST, metrics or stable public schema is added by this phase.

### Phase 2: MySQL end-to-end validation

Tracking: #11354 / #11364. Phase 2A implementation: [draft PR #12137](https://github.com/apache/seatunnel/pull/12137), dependent on #11512. It remains draft until the dependency and validation gates are met.

#### 2A. First bounded validation slice

- [x] Exercise real MySQL snapshot/binlog capture and a JDBC sink on separate embedded Zeta master/worker members.
- [x] Observe naturally collected reports through the internal coordinator service, including an enumerator hosted away from the master; do not inject reports or add a production observation endpoint.
- [x] Verify snapshot rows independently, consistent retained assignment counts, incremental insert/update/delete results, and native binlog advancement to a pre-batch database boundary.
- [x] Verify report identity, explicit unsupported checkpoint/restore facts, and removal of reports after cancellation.
- [x] Focused local MySQL execution passed twice on each of Java 8 and Java 11. Final clean runs also passed the existing engine RPC smoke test (2 executed, 0 failures/errors/skips per JVM).
- [ ] Complete broader engine classpath validation, supported CI, and dependency reconciliation before marking the draft ready. Two local passes per JVM are not a comprehensive stability campaign.

This slice proves one table and one reader under normal operation. Sink contents independently prove the full DML batch; the progress assertion proves consumption reaches at least one record from that batch. It does not prove every transient phase, per-record reporting, heartbeat behavior or recovery. Local Docker compatibility settings and exact validation evidence are recorded in the draft PR, separately from #11512's CI.

#### 2B. Snapshot detail and parallelism

- [ ] Validate two captured tables and parallel readers with correct source/table/split associations.
- [ ] Cover lazy chunk creation without inventing a final split total or percentage.
- [ ] Use deterministic test gates for transient snapshot states and split/watermark associations.
- [ ] Cover more than 100 active splits, truthful aggregate counts, bounded detail and explicit truncation.
- [ ] Cover empty tables and idle readers; agree on retained-count versus cumulative-total semantics after snapshot metadata is pruned.

#### 2C. Incremental progress and freshness

- [ ] Validate the real high-watermark transition from catch-up to incremental.
- [ ] Validate record/heartbeat positions and unchanged offsets without refreshing position-change time merely because sampling occurred.
- [ ] Distinguish fresh-but-unchanged, stale, missing and unavailable observations.
- [ ] Delay one reporting member and verify other members can still report.
- [ ] Keep no-progress observations separate from lag, backpressure and connector-proven stalls.

Exit: real data and reporting paths agree across normal, parallel and slow-reporting cases. Any product defect found is reproduced and scoped separately, not added silently to #11512.

### Phase 3: Checkpoint, restore and recovery observability

Tracking: this checklist under #11354 / #11364; no separate task issue.

- [ ] Agree on checkpoint identity/capabilities and explicit engine restore context before implementing lifecycle hooks.
- [ ] Capture checkpoint candidates at the actual snapshot callback; promote only on completion and discard aborted candidates.
- [ ] Validate checkpoint N completes, N+1 aborts, and the last completed position remains N; cover overlapping callbacks and retention bounds.
- [ ] Distinguish fresh submission, checkpoint restore and savepoint restore; ordinary split assignment is not evidence of restore.
- [ ] Verify restored origin, first post-restore consumption, new-attempt sequence ordering and rejection of late old-attempt reports.
- [ ] Coordinate with and reuse/extend the related cluster-failover harness in #11947; it is separate work, not a completed STIP-30 slice.
- [ ] Validate worker/master recovery, enumerator rediscovery, report cleanup and source/sink reconciliation together.

Exit: reported checkpoint/restore facts come from actual lifecycle evidence and recovery does not retain misleading old reports.

### Phase 4: PostgreSQL and multiple-source validation

Tracking: this checklist under #11354 / #11364; no separate task issue.

- [ ] Validate PostgreSQL LSNs through the real connector, reusing existing data/recovery tests where possible.
- [ ] Validate two CDC sources and separate pipelines without merging source identities, connector types or positions.
- [ ] Define internal freshness/partial aggregation without implying an atomic distributed snapshot.
- [ ] Preserve per-field exact, best-effort, unsupported and unavailable values, including non-provider connectors.
- [ ] Validate report serialization and native/opaque positions under the supported compatibility policy; do not introduce an unsupported mixed-version-cluster guarantee.

Exit: at least two native position families and multiple sources validate ownership, precision, nullability and cardinality before public-contract stabilization.

### Phase 5: Operator-facing API and metrics

Tracking: this checklist under #11354 / #11364; no separate task issue. Explicit interface agreement is required.

- [ ] Agree on the additive job-detail shape, capability matrix, payload limits and stabilization criteria.
- [ ] Add optional CDC detail without changing non-CDC responses or existing endpoints.
- [ ] Expose source/task identity, freshness, partial support and truncation with bounded serialized payloads.
- [ ] Add multi-source, missing-report, permission and compatibility contract tests.
- [ ] Add agreed low-cardinality metrics; never use native positions, table names or split IDs as unbounded labels.
- [ ] Review identifier/offset disclosure and sanitization; exclude credentials, connection strings and row values.
- [ ] Keep CLI/UI consumers optional follow-ups to the agreed API, not a separate schema or release commitment.

Exit: truthful, backward-compatible operator output. Public/schema stabilization waits for Phases 2–4 and the applicable Phase 6 gates; no inferred ETA or universal lag guarantee.

### Phase 6: Cross-cutting measurement, stability and adoption

Tracking: this checklist under #11354 / #11364; no separate task issue. Start alongside Phase 2, not after freezing the public contract.

- [ ] Compare baseline and feature costs for idle, high-throughput, many-table, high-parallelism and checkpoint-heavy workloads.
- [ ] Measure CPU/allocation, retained memory, report bytes, collection latency and checkpoint latency; agree budgets from evidence.
- [ ] Test repeated start/stop and failover for bounded state and resource cleanup.
- [ ] Run affected Java 8/11 suites and supported CI environments; separate environment limitations from product failures.
- [ ] Update English/Chinese operational documentation with capabilities and troubleshooting examples.
- [ ] Record explicit community stabilization acceptance and agreed release scope before changing the STIP to Accepted/Implemented as appropriate.

Exit: measured, documented and accepted behavior, not merely a set of implementation PRs.

### Updating this tracker

Keep this issue canonical. Link a task issue or PR when one actually exists, update checkboxes only from review/test/merge evidence, and keep dependency, draft and approval status explicit. Before each new slice, recheck current code, related work and maintainer direction. No new issue is needed for each checkbox.

## Acceptance criteria

- reporting does not block record processing or checkpoint callbacks
- reader reports are collected on execution members; enumerator discovery and collection are coordinator-owned, with reports requested from the hosting member
- lifecycle events can request refreshes but correctness is rebuilt from current reader/enumerator state
- report ordering includes execution attempt identity and accepts a restarted task whose sequence begins at zero
- snapshot and watermark values are associated through the same split
- current, checkpoint candidate, completed checkpoint, and restored positions remain distinct
- temporary cross-task inconsistency is visible through freshness metadata
- lifecycle, recovery, health, and backpressure are not conflated
- unsupported and best-effort fields are explicit
- multiple CDC source vertices remain separate in job detail
- native positions, split IDs, and table paths are never metric labels
- job-detail payload is latest-only and bounded
- MySQL and PostgreSQL validate the contract before public API stabilization
- documentation explains normalized fields, native positions, accuracy, and connector limitations

### Usage Scenario

This is for operators running long-lived CDC jobs who need to answer:

- Is the source still snapshotting, catching up, or consuming incrementally?
- Which active reader or split is not advancing?
- What position was last completed in a checkpoint?
- Did the job restore from the expected position?
- Is a report stale, is no progress observed, or does the connector have direct evidence of a stalled condition?
- Which values are exact, best effort, or unsupported for this connector?

### Related issues

#11354

### Are you willing to submit a PR?

- [x] Yes, I am willing to submit a PR.

### Code of Conduct

- [x] I agree to follow the Apache Software Foundation Code of Conduct.