Add a config to allow partition column type inference in bootstrap
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, we assume that the partition column is always in String type during bootstrap operation. TestDataSourceForBootstrap.testMetadataBootstrapCOWHiveStylePartitioned fails for date partition column if the type inference of partition column is turned on.
We need to add a config to allow partition column inference in bootstrap so that other types of partition columns are supported.
HoodieSparkBootstrapSchemaProvider
{code:java}
private static Schema getBootstrapSourceSchemaParquet(HoodieWriteConfig writeConfig, HoodieEngineContext context, Path filePath) {
// NOTE: The type inference of partition column in the parquet table is turned off explicitly,
// to be consistent with the existing bootstrap behavior, where the partition column is String
// typed in Hudi table.
((HoodieSparkEngineContext) context).getSqlContext()
.setConf(SQLConf.PARTITION_COLUMN_TYPE_INFERENCE(), false);
StructType parquetSchema = ((HoodieSparkEngineContext) context).getSqlContext().read()
.option("basePath", writeConfig.getBootstrapSourceBasePath())
.parquet(filePath.toString())
.schema(); {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4932
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1265
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieSparkBootstrapSchemaProvider.getBootstrapSourceSchemaParquet and the failing TestDataSourceForBootstrap.testMetadataBootstrapCOWHiveStylePartitioned test. Trace how bootstrap configuration is defined and passed into Spark SQL, then verify that the new setting controls partition-column inference and that date partition columns pass the metadata bootstrap test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100