Fix AdaptiveSparkPlanExec wrapped by ColumnarToCarrierRow breaks shuffle IDs retrieval
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
## **Description**
Gluten's columnar writer optimization wraps `AdaptiveSparkPlanExec` with `ColumnarToCarrierRow` to avoid unnecessary columnar-to-row conversions. However, this breaks the pattern matching used in Apache Spark PR #51432, which relies on:
```scala
queryExecution.executedPlan match {
case ae: AdaptiveSparkPlanExec =>
ae.context.shuffleIds.asScala.keys
}
```
When `AdaptiveSparkPlanExec` is wrapped by `ColumnarToCarrierRow`, the pattern matching fails, making shuffle IDs inaccessible.
### **Root Cause**
In `GlutenWriterColumnarRules.injectFakeRowAdaptor()`, when the child is an `AdaptiveSparkPlanExec`, the original implementation:
1. Created a new `AdaptiveSparkPlanExec` with `supportsColumnar=true`
2. Wrapped this with `genColumnarToCarrierRow()` → `ColumnarToCarrierRow(AdaptiveSparkPlanExec(...))`
This structure hides `AdaptiveSparkPlanExec` inside `ColumnarToCarrierRow`, breaking any external pattern matching.
### **Solution**
Refactored the wrapping logic to:
1. Wrap `aqe.inputPlan` with `genColumnarToCarrierRow()` first → `ColumnarToCarrierRow(inputPlan)`
2. Create a new `AdaptiveSparkPlanExec` with the wrapped child → `AdaptiveSparkPlanExec(ColumnarToCarrierRow(...))`
3. Set `supportsColumnar=false` since the child is already wrapped
### Gluten version
main branch
### Spark version
spark-4.0.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Assessment
This issue has not been assessed yet.