apache / apache/gluten

[VL][Delta] Add UniForm Iceberg support for Delta tables

Open
#12,039 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

## Summary

Add UniForm Iceberg support for Delta tables on the Velox backend.

Today Gluten documents `Iceberg readers (UniForm)` as `Not tested` in `docs/get-started/VeloxDelta.md`. This issue tracks enabling and validating the supported Velox path.

## Current state

Gluten already has some relevant hooks:

- `backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/GlutenOptimisticTransaction.scala`
- materializes partition columns for Iceberg compat
- tags `AddFile` entries with Iceberg compat version
- `backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/GlutenDeltaParquetFileFormat.scala`
- forces `TIMESTAMP_MICROS`
- switches to `DeltaParquetWriteSupport` for `IcebergCompatV2`

During investigation we also confirmed that Velox already supports explicit Parquet `field_id` assignment in its native Parquet writer.

## Main gaps

- no `delta-iceberg` dependency/test enablement in Gluten build/test flow
- no end-to-end Velox UniForm test coverage
- no proof that native Delta write passes the nested field IDs required by `IcebergCompatV2` into the native Parquet writer
- no explicit validation/fallback for unsupported cases such as active deletion vectors
- no supported-scope documentation beyond `Not tested`

## Relevant Delta requirements

At minimum, the supported path needs:

- column mapping enabled
- `minReaderVersion >= 2`
- `minWriterVersion >= 7`
- `delta.enableIcebergCompatV2=true`
- `delta.universalFormat.enabledFormats=iceberg`
- Delta 3.1+ writer
- Hive Metastore-backed Iceberg catalog path
- no active deletion vectors
- partition columns materialized in Parquet
- `numRecords` populated in new `AddFile` stats
- timestamp columns written as int64 / micros
- nested array/map field IDs written into the Parquet schema

## References

- Delta UniForm docs: https://docs.delta.io/delta-uniform/
- Delta protocol: https://github.com/delta-io/delta/blob/master/PROTOCOL.md

Contributor guide

Open the contributing guide

Research direction

Start with the Velox Delta hooks in GlutenOptimisticTransaction.scala and GlutenDeltaParquetFileFormat.scala, then inspect the Gluten build/test flow for delta-iceberg enablement. Validate the end-to-end UniForm path against the listed Delta requirements, including nested field IDs and unsupported deletion vectors. Done means coverage exists and docs/get-started/VeloxDelta.md no longer marks Iceberg readers as Not tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, build-system, databases, documentation, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.