apache / apache/hudi

[SUPPORT]Unable to read new data in streaming mode with specific timestamp

Open
#12,661 7 comments 0 reactions 0 assignees View on GitHub
engine:flink
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
When using Flink to read a Hudi COW table in streaming mode, specific timestamp can read new data written after the Flink job starts. The streaming job only reads data up to its start time.

**To Reproduce**
```java
Map options = new HashMap<>();
options.put(FlinkOptions.PATH.key(), basePath + tableName);
options.put(FlinkOptions.TABLE_TYPE.key(), HoodieTableType.COPY_ON_WRITE.name());
options.put(FlinkOptions.READ_AS_STREAMING.key(), "true");

// specific timestamp - same behavior
options.put(FlinkOptions.READ_START_COMMIT.key(), "20240116000000");

HoodiePipeline.Builder builder = HoodiePipeline.builder(tableName)
.options(options);
DataStream rowDataDS = builder.source(env);
```

**Expected behavior**
The streaming job should continuously read new data written after job starts, regardless of using specific timestamp.

**Environment Description**
* Hudi version: 0.14.0
* Flink version: 1.16.0
* Hadoop version: 3.1.0
* Storage: HDFS

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at HoodiePipeline.Builder.source(env) and trace how READ_AS_STREAMING and READ_START_COMMIT are handled for a Flink COW table. Reproduce with the options and environment described, then verify that the job continues reading data written after startup when a specific timestamp is supplied.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.