[VL] Move Iceberg partition transform from TableWriter node to a new Projection node
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Currently, Iceberg partition transform is performed in the execution layer (within TableWrite operator). For better architecture and optimization opportunities, this computation should be moved to the optimizer/planner layer where it logically belongs. And this can be achieved by adding a projection node above TableWriter node and letting this node to perform the partition transform and send the pre-computed partition keys to TableWriter.
Doing so would allow for this computation to be optimized further by, for example, pushing it below the exchange and letting it run with higher concurrency than write itself. It would also provide more transparency for debugging as we would get runtime stats just for this piece of computation. It would allow users to see more clearly in the query plan what is happening allowing for more efficient debugging.
### Gluten version
None
CC @mbasmanova @jinchengchenghh @zhouyuan
Contributor guide
Research direction
Inspect the execution-layer TableWriter and the optimizer/planner components involved in Iceberg partition transforms. Determine how a new projection above TableWriter would pass pre-computed partition keys, and verify that the completed plan exposes this computation separately for optimization and runtime statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100