[SUPPORT]Unable to read new data in streaming mode with specific timestamp
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
When using Flink to read a Hudi COW table in streaming mode, specific timestamp can read new data written after the Flink job starts. The streaming job only reads data up to its start time.
**To Reproduce**
```java
Map options = new HashMap<>();
options.put(FlinkOptions.PATH.key(), basePath + tableName);
options.put(FlinkOptions.TABLE_TYPE.key(), HoodieTableType.COPY_ON_WRITE.name());
options.put(FlinkOptions.READ_AS_STREAMING.key(), "true");
// specific timestamp - same behavior
options.put(FlinkOptions.READ_START_COMMIT.key(), "20240116000000");
HoodiePipeline.Builder builder = HoodiePipeline.builder(tableName)
.options(options);
DataStream rowDataDS = builder.source(env);
```
**Expected behavior**
The streaming job should continuously read new data written after job starts, regardless of using specific timestamp.
**Environment Description**
* Hudi version: 0.14.0
* Flink version: 1.16.0
* Hadoop version: 3.1.0
* Storage: HDFS
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at HoodiePipeline.Builder.source(env) and trace how READ_AS_STREAMING and READ_START_COMMIT are handled for a Flink COW table. Reproduce with the options and environment described, then verify that the job continues reading data written after startup when a specific timestamp is supplied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100