Gluten consumes much higher memory and created much larger bloomfilter subquery datasize than Spark
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Noted as a known issue here. The issue happens when you set **large creationSideThreshold, maxNumBits, and maxNumItems with CBO enabled**.
When CBO (Cost-Based Optimizer) is disabled, Spark creates a Bloom filter subquery based on creationSideThreshold, falling back to default values for estimatedNumItems and numBits. The query successfully runs with partial_bloom_filter_agg(expr, estimatedNumItems, numBits, 0, 0).
However, when CBO is enabled, Spark estimates estimatedNumItems and numBits dynamically. If these CBO-estimated values become exceptionally large, partial_velox_bloom_filter_agg will consume significantly more memory and generate a much larger Bloom filter compared to native Spark under the exact same configuration.
This discrepancy leads to the following cascading failures:
Gluten throws a spark.rpc.message.maxSize error due to the massive payload size.
Even if spark.rpc.message.maxSize is significantly increased, the subsequent velox_bloom_filter_agg operator eventually throws an OOM (Out of Memory) error.
e.g. if we test TPCDS SF1T, the ```partial_bloom_filter_agg(xxhash64(wr_order_number#65L, 42), 72002305, 739372223, 0, 0)``` can pass on Spark but OOM on gluten.
The shuffle data size on Spark is only **69.4M**, while on Gluten it's **9.3 GiB**!
cc @jinchengchenghh
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with the VL (Velox) backend and the partial_velox_bloom_filter_agg operator, then reproduce the TPCDS SF1T example with CBO enabled and large creationSideThreshold, maxNumBits, and maxNumItems. Compare Spark and Gluten shuffle sizes, payload limits, and memory usage; done means the Gluten path no longer produces the reported 9.3 GiB payload or OOM under the described conditions.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100