apache / apache/hudi

Revisit dropped Partition Columns handling

Open
#15,407 0 comments 0 reactions 0 assignees View on GitHub
area:writer from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, dropping partition columns (controlled by "hoodie.datasource.write.drop.partition.columns") is handled in a piecemeal fashion, which unfortunately may to lead to very subtle and hard to troubleshoot issues when used.

For ex, currently in HoodieSparkSqlWriter this would affect what will be persisted as writer's schema – in case partition columns are dropped from the data file we will persist "reduced" schema as the one that was used by the Writer, which is invalid since Writer was using the full schema, however partition columns weren't persisted in the Data Files (ie dropped, since they're already encoded into the partition path)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4772
- Type: Bug
- Affects version(s):
- 0.13.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at HoodieSparkSqlWriter and trace how the hoodie.datasource.write.drop.partition.columns setting affects the writer schema and persisted data-file schema. Done means the writer uses the full schema while partition columns remain omitted from data files and are represented in the partition path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.