apache / apache/hudi

Add a config to allow partition column type inference in bootstrap

Open
#15,459 1 comment 0 reactions 0 assignees View on GitHub
component:bootstrap from-jira priority:high status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, we assume that the partition column is always in String type during bootstrap operation.  TestDataSourceForBootstrap.testMetadataBootstrapCOWHiveStylePartitioned fails for date partition column if the type inference of partition column is turned on.

 

We need to add a config to allow partition column inference in bootstrap so that other types of partition columns are supported.

 

HoodieSparkBootstrapSchemaProvider
{code:java}
private static Schema getBootstrapSourceSchemaParquet(HoodieWriteConfig writeConfig, HoodieEngineContext context, Path filePath) {
// NOTE: The type inference of partition column in the parquet table is turned off explicitly,
// to be consistent with the existing bootstrap behavior, where the partition column is String
// typed in Hudi table.
((HoodieSparkEngineContext) context).getSqlContext()
.setConf(SQLConf.PARTITION_COLUMN_TYPE_INFERENCE(), false);
StructType parquetSchema = ((HoodieSparkEngineContext) context).getSqlContext().read()
.option("basePath", writeConfig.getBootstrapSourceBasePath())
.parquet(filePath.toString())
.schema(); {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4932
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1265

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HoodieSparkBootstrapSchemaProvider.getBootstrapSourceSchemaParquet and the failing TestDataSourceForBootstrap.testMetadataBootstrapCOWHiveStylePartitioned test. Trace how bootstrap configuration is defined and passed into Spark SQL, then verify that the new setting controls partition-column inference and that date partition columns pass the metadata bootstrap test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.