apache / apache/paimon

[Bug] The scan.timestampmillis parameter is not effective

Open
#5,720 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon 0.9

### Compute Engine

flink 1.18.1

### Minimal reproduce step

Stream read the Append table and write data to two Aggregate tables, one of which is a partitioned table and the other is a non partitioned table. If the scan.timestampmillis parameter set by the Append table data source does not take effect, the data will be consumed from historical data instead of the set point (if written separately to one of the tables, the parameter is valid)

### What doesn't meet your expectations?

The parameters set by scan.timestampmillis should take effect, starting from the set point and regardless of the table being written.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with Paimon 0.9 and Flink 1.18.1 by reading an Append table into both partitioned and non-partitioned Aggregate tables. Check whether scan.timestampmillis starts consumption at the configured point in both cases; done means neither Aggregate table reads historical data before that point.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.