[Bug] The configurations of `second-to-micro` and `millis-to-micro` within `sequence.auto-padding` does not meet the expected behavior.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Paimon version
<=master
### Compute Engine
flink <=master
### Minimal reproduce step
The configurations of `second-to-micro` and `millis-to-micro` within `sequence.auto-padding` can lead to issues with nanotime generation (similar to data overflow). This can result in a combined sequence in later incoming data being smaller than that in historical data when they share the same `sequence.field`. Consequently, this might lead to the data being discarded, even though, theoretically, incoming data should always be the latest, provided they share the same `sequence.field`.
### What doesn't meet your expectations?
See https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1670885102.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the sequence.auto-padding configuration and the linked PR comment describing the second-to-micro and millis-to-micro behavior. Reproduce the nanotime or overflow case and determine the expected ordering for incoming and historical records sharing a sequence.field; done means incoming records are not discarded because of a smaller combined sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100