apache / apache/hudi

Flesh out schema evolution support for variant

Open
#18,285 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**

https://github.com/apache/hudi/pull/17833/changes#r2874744476

https://github.com/apache/hudi/pull/17833#discussion_r2957128901

**Why this task is needed:**

Comment from @vinothchandar
What kind of evolution are we able to support with Variant columns? Can we ensure there are tests around.

- Adding a Variant column to an existing table
- Removing a Variant column
- `maxColumnId` calculation when Variant fields (with negative IDs) coexist with regular fields
- Round-trip: `HoodieSchema → InternalSchema → HoodieSchema` for Variant types

### Additional consideration: variant projection vs schema-evolution ordering (from PR #18923)

PR #18923 moved Spark 4.1 PushVariantIntoScan projection out of the engine-neutral merge buffer into the log readers (parquet reader native projection, avro `deserializeRecords`). This flips the order of schema-evolution projection and variant projection in the MOR read path:

- Before: evolve then project. `composeEvolvedSchemaTransformer` ran against full VariantType rows (consistent with `dataBlock.getSchema()`), then variant projection rewrote them.
- After: project then evolve. Rows reach the buffer already projected, but `composeEvolvedSchemaTransformer` still builds its projector with `from = dataBlock.getSchema()` (variant typed as VariantType). When `internalSchema` is non-empty, the evolve step would mis-decode the projected struct bytes as VariantVal.

This only triggers with the (schema-on-read active + variant column + PushVariantIntoScan + MOR log blocks) combination, which is the same unsupported matrix this task covers. When designing variant schema evolution, make `composeEvolvedSchemaTransformer` aware that variant fields may already be projected (build the evolve step against the projected schema) or otherwise reconcile the ordering. Refs: `FileGroupRecordBuffer.composeEvolvedSchemaTransformer`, `InternalSchemaConverter` line 510.

Exact trigger: both `internalSchema` is non-empty AND `shouldProjectVariants` is true (Spark only). If either is false there is no mismatch. With an empty `internalSchema` the evolve step is identity (`composeEvolvedSchemaTransformer` returns empty), so rows pass through untouched. With `shouldProjectVariants` false the rows are never projected upstream (both the avro `projectLogBlockRecords` and the parquet `getUnsafeRowIterator` overload are gated on it), so the evolve step sees consistent full VariantType rows.

Guard placement caveat: a fail-fast guard is awkward to add today because `internalSchema` lives in the engine-neutral buffer while `shouldProjectVariants` lives in the Spark reader context. The buffer cannot see the variant gate directly, so a clean guard would need a small reader-context hook (e.g. `readerContext.hasPendingVariantProjection()`) checked alongside the non-empty `internalSchema`.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with FileGroupRecordBuffer.composeEvolvedSchemaTransformer and InternalSchemaConverter around line 510, then review the linked PR discussions and the Spark reader paths for variant projection. Verify coverage for adding and removing Variant columns, maxColumnId handling, and HoodieSchema → InternalSchema → HoodieSchema round-trips. Done means schema evolution and variant projection work together for the described MOR read-path combination.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.