apache / apache/incubator-xtable

Add Hudi table version 9 (Hudi 1.x) support to the Hudi target

Open
#834 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

## Background

Part of the Hudi 1.x upgrade (#762).

To keep #772 a clean binary upgrade, the Hudi target currently **pins writes to table version 6** (the old timeline view) and **disables auto-upgrade**, so the write client does not upgrade tables to version 9:

- `HudiTableManager.initializeHudiTable`: `.setTableVersion(HoodieTableVersion.SIX)`
- `HudiConversionTarget` write config: `.withWriteTableVersion(HoodieTableVersion.SIX.versionCode())` and `.withAutoUpgradeVersion(false)`

This preserves backward compatibility with the pre-upgrade behavior (instants selected and ordered by requested/instant time, column-stats index version V1).

## Upstream status: the two Hudi blockers are now merged

Both upstream fixes this issue was waiting on have landed on Hudi master:

- **apache/hudi#19111** (merged 2026-06-30) reintroduces `hoodie.metadata.index.partition.stats.enable` as an advanced config, so the partition-stats index can be disabled independently of column stats. `isPartitionStatsIndexEnabled()` is now `columnStatsEnabled && partitionStatsEnabled`, defaulting to `true` so existing behaviour is unchanged. This matters directly for XTable: partition-stats generation rebuilds a `FileSystemView` over the base files and is not supported for externally created files, so before this change the only workaround was disabling column stats entirely and losing file-level data skipping. We can now keep column stats on and turn partition stats off.
- **apache/hudi#19126** (merged 2026-07-23) fixes a crash when building the column-stats index on a **table version 9** (timeline layout V2) Merge-on-Read table where the col-stats column list contains a leaf nested inside a `MAP` or `ARRAY`, for example `my_map.key_value.value`. It previously failed during the inline log-append upsert with `IllegalStateException: Cannot get field from schema type: MAP`.

## Remaining gate: no released Hudi contains them

Neither fix is in a Hudi release yet.

- The latest Hudi 1.x release is **1.2.0, published 2026-05-23**, which predates both merges. Both merge commits are `diverged` from the `release-1.2.0` branch, and there is no `release-1.3.0` branch yet.
- XTable's `main` pins `hudi.version` to **1.2.0**.

So this work can be implemented and verified against a locally built Hudi master, but it cannot ship until XTable can depend on a Hudi release carrying both fixes. That dependency bump should be tracked as part of this issue.

## Known limitation at table version 6 (index V1): decimal and fixed column stats

At column-stats index **V1**, Hudi's `HoodieTableMetadataUtil.isColumnTypeSupported` (the `isColumnTypeSupportedV1` branch, when no record type is supplied) explicitly excludes `DECIMAL`, `FIXED` and `BYTES` columns, since DECIMAL's underlying type is BYTES. This is tracked upstream as HUDI-8585, which is **still open** as of 2026-08-10 and remains in the code on Hudi master. Note that we do not need it: moving to index V2 is itself the fix, so this is a reason to do this issue rather than a dependency of it. The same gate is used both by the MDT col-stats writer (`getColumnsToIndex`) and by XTable's parquet-footer fallback (`HudiFileStatsExtractor.computeColumnStatsForFile`).

Consequence on table version 6: a decimal or `fixed` column gets **no column stats**, even though the parquet footer contains valid min and max values for it. Index **V2** (table version 9) does support these types, since `isColumnTypeSupportedV2` does not exclude DECIMAL, FIXED or BYTES.

This is why `TestHudiFileStatsExtractor` currently asserts 8 of 9 columns, dropping the decimal column. All of its tables and mocks are pinned to version 6.

## Scope

- Allow writing table version 9, the new MDT and timeline layout, behind a feature flag so users can opt in.
- Switch instant selection and ordering to **completion time** when on version 9, with tests covering out-of-order completion. See the discussion on `HudiConversionSource`.
- Set the partition-stats config from apache/hudi#19111 for XTable-written tables: column stats on, partition stats off, since partition stats are not supported over externally created files.
- **Parameterize the col-stats tests, for example `TestHudiFileStatsExtractor`, over table version 6 and 9.** Version 6 with index V1 drops the decimal column, giving 8 columns. Version 9 with index V2 keeps it, giving 9 columns. This replaces the current v6-only assertions.
- Cover the decimal and fixed value standardization paths, since the column-stats index moves from V1 to V2 at table version 9.
- Bump `hudi.version` to the first Hudi release that contains apache/hudi#19111 and apache/hudi#19126, and remove the version-6 pin once v9 is the default.

## Related

- #832, enable the MDT column-stats index for partitioned tables, which depends on the same partition-stats config.
- #762, the Hudi version upgrade this belongs to.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HudiTableManager.initializeHudiTable, HudiConversionTarget's write configuration, and the HudiConversionSource discussion to trace version and instant-ordering behavior. Run TestHudiFileStatsExtractor and inspect the col-stats tests, then verify version 6 and 9 expectations, decimal and fixed standardization, the partition-stats configuration, and the required Hudi dependency bump against a release containing both upstream fixes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.