apache / apache/hudi

For hive-style partitioned source data, partition columns synced with Hive will always have String type

Open
#14,599 0 comments 0 reactions 0 assignees View on GitHub
component:bootstrap component:catalog-sync from-jira priority:blocker type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently bootstrap implementation is not able to handle partition columns correctly when the source data has *hive-style partitioning*, as is also mentioned in https://jira.apache.org/jira/browse/HUDI-915

The schema inferred while performing bootstrap and stored in the commit metadata does not have partition column schema(in case of hive partitioned data). As a result during hive-sync when hudi tries to determine the type of partition column from that schema, it would not find it and assume the default data type *string*.

Here is where partition column schema is determined for hive-sync:

[https://github.com/apache/hudi/blob/master/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveSchemaUtil.java#L417]

 

Thus no matter what the data type of partition column is in the source data (atleast what spark infers it as from the path), it will always be synced as string.

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-992
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-1265
- Affects version(s):
- 0.9.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HiveSchemaUtil.java around line 417, then trace the bootstrap schema stored in commit metadata for hive-style partitioned source data. Verify how Hive sync determines partition types and confirm that the synced type preserves the source or Spark-inferred type instead of defaulting to String.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.