Avoid duplicate bucket ID projection in native write paths
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
## Overview
Native write paths in Spark 3.2/3.3 and the ClickHouse MergeTree writer
recompute the `__bucket_value__` expression even when a precomputed
attribute already exists. This adds unnecessary overhead and complicates
downstream projections.
## Steps to Reproduce
- Write bucketed data using the native writer.
- Inspect the execution plan; the bucket ID is projected multiple times
instead of reusing a single attribute.
## Expected Behavior
The bucket ID should be computed once (e.g., in an initial `ProjectExec`)
and reused by subsequent stages.
## Actual Behavior
Every stage re-evaluates the bucket expression, leading to redundant
projections and performance overhead.
## Impact
- Increased CPU time for bucketed writes.
- Harder-to-read execution plans with repetitive projections.
## Proposed Fix
- Guard projections in Spark 3.2/3.3 shims and ClickHouse MergeTree writer
so they append `__bucket_value__` only when missing.
- Store the computed bucket ID in an attribute and reuse it downstream.
Contributor guide
Research direction
Start by reading the Spark 3.2/3.3 shims and the ClickHouse MergeTree writer, then reproduce the native bucketed write and inspect its execution plan. Trace where __bucket_value__ is projected and compare the paths that already have a precomputed attribute. Done means the bucket expression is projected once and reused downstream without duplicate projections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala, spark
- Domain
- data-engineering, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100