apache / apache/hudi

[SUPPORT] Partition field value lost in table column

Open
#7,242 4 comments 0 reactions 0 assignees View on GitHub
component:catalog-sync priority:medium release-0.12.2 type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am running spark-submit job to populate data into hudi tables from kafka topics. I have below properties set in my table-config.properties file:
hoodie.datasource.write.partitionpath.field=partitionFieldColumn
hoodie.datasource.hive_sync.table=tabledata
hoodie.datasource.hive_sync.partition_fields=partitionFieldColumn
hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS
hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd

I am using "partitionFieldColumn" which is of datetime type. Using this, I want to have 3 level of partitioning (year -> month -> date). To avoid using time in the partitioning, "hoodie.deltastreamer.keygen.timebased.output.dateformat" property has the date format value. This results in correct partitioning levels but the "partitionFieldColumn" column created in the "tabledata" table also has the time field truncated, which is data loss.

Is there any way to retain the complete value of the "partitionFieldColumn" in the hive table without truncating the time field?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with the listed table-config.properties settings and a datetime partitionFieldColumn in the spark-submit ingestion flow. Compare the partition paths with the corresponding tabledata column value, then investigate how the timebased key generator and Hive sync handle the formatted partition field. Done means date-based partitions remain while the table column retains the complete datetime value.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.