apache / apache/paimon

[Bug] `CdcWatermarkStrategy` The current system time will always be sent as the watermark value.

Open
#2,745 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

master

### Compute Engine

flink

### Minimal reproduce step

Appears in any case, as long as watermark is generated.
`onPeriodicEmit` is called periodically, so the system timestamp will be periodically sent out as a watermark. This is not what we expect.

https://github.com/apache/incubator-paimon/blob/1bc33d9f606e84c2341cea4e604ce17d6651921f/paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/watermark/CdcWatermarkStrategy.java#L63

### What doesn't meet your expectations?

Watermark should be generated based on the time in the data, here all are `system timestamp`

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with onPeriodicEmit in paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/watermark/CdcWatermarkStrategy.java at the referenced line, then trace how event-time values are available to the strategy. The fix is done when generated watermarks use timestamps from the data rather than the current system time.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.