apache / apache/hudi

Date columns showing incorrect integer values while doing CDC read

Open
#16,408 1 comment 0 reactions 0 assignees View on GitHub
area:incr-processing from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Date columns are showing integer values while doing CDC read.

For timestamp columns also it is converting into epoch , may be doing similar conversion for date column causing wrong integer values. 

 

Reproducible Code - 

 

```
columns = ["ts","uuid","rider","driver","fare","city", "report_date"]
data =[(1695159649087,"334e26e9-8355-45cc-97c6-c31daf0df330","rider-A","driver-K",19.10,"san_francisco", "2022-01-01"),
(1695091554788,"e96c4396-3fad-413a-a942-4cb36106d721","rider-B","driver-L",27.70 ,"san_francisco", "2022-01-01"),
(1695046462179,"9909a8b1-2d15-4d3d-8ec9-efc48c536a00","rider-C","driver-M",33.90 ,"san_francisco", "2022-01-01"),
(1695516137016,"e3cf430c-889d-4015-bc98-59bdce1e530c","rider-C","driver-N",34.15,"sao_paulo", "2022-01-01")]

inserts = spark.createDataFrame(data).toDF(*columns)

hudi_options =

{ 'hoodie.table.name': tableName, 'hoodie.datasource.write.recordkey.field' : 'uuid', 'hoodie.datasource.write.precombine.field' : 'ts', 'hoodie.datasource.write.partitionpath.field': 'city', 'hoodie.datasource.write.reconcile.schema':'true', 'hoodie.table.cdc.enabled':'true', }

inserts = inserts.withColumn("report_date", expr("CAST(report_date as timestamp)"))

inserts.write.format("hudi"). \
options(**hudi_options). \
mode("overwrite"). \
save(basePath)

cdc_read_options = \{ 'hoodie.datasource.query.incremental.format': 'cdc', 'hoodie.datasource.query.type': 'incremental', 'hoodie.datasource.read.begin.instanttime': 0 }

df = spark.read.format("hudi").load(basePath)
df.show(10, False)

spark.read.format("hudi"). \
options(**cdc_read_options). \
load(basePath).show(10, False)
```

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-7461
- Type: Bug
- Fix version(s):
- 0.16.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied Spark reproduction and compare the regular Hudi read with the CDC read, focusing on the report_date and timestamp columns. Trace the CDC date and timestamp conversion path used by the incremental read. Done means date values remain readable dates and timestamp values are not incorrectly represented as epoch integers, with the reproduction covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.