Support completion time as checkpoint in S3Event/GcsEvent incremental source in Hudi Streamer
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, S3Event/GcsEvent source uses `QueryInfo` to get time range for incremental queries.
`QueryContext` uses completion time to get time range for incremental queries and is more ideal. But QueryContext misses some logic that are required by S3/GCSEventSource. These logic can be found under IncrSourceHelper#generateQueryInfo: https://github.com/apache/hudi/blob/622c8a37cafea903b59c66fd4d40c80380df8e94/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L158
The goal is to:
1. migrate these logic to QueryContext
2. Have S3/GCSEvent source uses QueryContext instead
3. Remove QueryInfo
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8354
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9105
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.