apache / apache/hudi

[SUPPORT] _hoodie_commit_timestamp showing NULL values for some rows when performing a streaming query using Flink SQL

Open
#12,978 6 comments 0 reactions 0 assignees View on GitHub
engine:flink priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? y

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

`_hoodie_commit_timestamp` is NULL for some rows while performing a streaming query using Flink SQL. I have also tried this using the Flink Table API for java, and the issue seems to be prevalent there as well. I am demonstrating the issue in Flink SQL since it is easier to reproduce.

**To Reproduce**

Steps to reproduce the behavior:

1. Run these SQL commands in the flink SQL bash script of the jobmanager in one shell. (lets call this shell 1)
```
CREATE TABLE hudi_table (
_hoodie_commit_time STRING,
uuid STRING PRIMARY KEY NOT ENFORCED, -- Record key
test BOOLEAN -- Check column
)
WITH (
'connector' = 'hudi',
'path' = 's3a://test-bucket/test', -- Path to your Hudi table
'table.type' = 'COPY_ON_WRITE', -- Specify the table type (COW or MOR)
'cdc.enabled' = 'true', -- Enable CDC
'hoodie.datasource.write.recordkey.field' = 'uuid' -- Record key field
);
```
2. In the second SQL client shell Run these commands (Lets call this shell 2)
```
CREATE TABLE hudi_table (
_hoodie_commit_time STRING,
uuid STRING PRIMARY KEY NOT ENFORCED, -- Record key
test BOOLEAN -- Check column
)
WITH (
'connector' = 'hudi',
'path' = 's3a://test-bucket/test', -- Path to your Hudi table
'table.type' = 'COPY_ON_WRITE', -- Specify the table type (COW or MOR)
'cdc.enabled' = 'true', -- Enable CDC
'hoodie.datasource.write.recordkey.field' = 'uuid' -- Record key field
);
select * from hudi_table/*+ OPTIONS('read.streaming.enabled'='true')*/;
```
The output currently after running this command is

3. In shell 1 run these commands (give some time before running the second insert command)
```
INSERT INTO hudi_table (uuid, test) VALUES ('1', true);
-- Wait for the row to be reflected in shell 1 output (This one has the _hoodie_commit_timestamp)
-- When it appears run the below INSERT command
INSERT INTO hudi_table (uuid, test) VALUES ('4', true);
```
4. Output in shell 2 is shown below
```
SQL Query Result (Table)
Refresh: 1 s Page: Last of 1 Updated: 08:08:50.900

_hoodie_commit_time uuid test
20250314080450519 1 TRUE
4 TRUE

Q Quit + Inc Refresh G Goto Page N Next Page O Open Row
R Refresh - Dec Refresh L Last Page P Prev Page
```
**Expected behavior**

`_hoodie_commit_timestamp` should not be null for all rows regardless of which query I use or what read.start-commit value I use. Right now, it appears to be NULL for some rows.

**Environment Description**

* Hudi version : 0.15.0

* Spark version :

* Hive version :

* Hadoop version : 3.3.4

* Storage (HDFS/S3/GCS..) : Minio (S3)

* Running on Docker? (yes/no) : yes

**Additional context**

I am using docker for flink, and have added the required jars to the lib folder for hudi.

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the two-shell Flink SQL reproduction against Hudi 0.15.0 with the shown COPY_ON_WRITE, CDC, and streaming-read options. Compare the commit metadata for the first and subsequent inserted rows, and check the Flink Table API path as well. Done means every streamed row exposes a non-NULL commit timestamp or the issue documents the supported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.