apache / apache/gluten

[VL] Move Iceberg partition transform from TableWriter node to a new Projection node

Open
#11,023 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Currently, Iceberg partition transform is performed in the execution layer (within TableWrite operator). For better architecture and optimization opportunities, this computation should be moved to the optimizer/planner layer where it logically belongs. And this can be achieved by adding a projection node above TableWriter node and letting this node to perform the partition transform and send the pre-computed partition keys to TableWriter.

Doing so would allow for this computation to be optimized further by, for example, pushing it below the exchange and letting it run with higher concurrency than write itself. It would also provide more transparency for debugging as we would get runtime stats just for this piece of computation. It would allow users to see more clearly in the query plan what is happening allowing for more efficient debugging.

### Gluten version

None

CC @mbasmanova @jinchengchenghh @zhouyuan

Contributor guide

Open the contributing guide

Research direction

Inspect the execution-layer TableWriter and the optimizer/planner components involved in Iceberg partition transforms. Determine how a new projection above TableWriter would pass pre-computed partition keys, and verify that the completed plan exposes this computation separately for optimization and runtime statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.