apache / apache/gluten

Fix AdaptiveSparkPlanExec wrapped by ColumnarToCarrierRow breaks shuffle IDs retrieval

Open
#11,752 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

## **Description**

Gluten's columnar writer optimization wraps `AdaptiveSparkPlanExec` with `ColumnarToCarrierRow` to avoid unnecessary columnar-to-row conversions. However, this breaks the pattern matching used in Apache Spark PR #51432, which relies on:

```scala
queryExecution.executedPlan match {
case ae: AdaptiveSparkPlanExec =>
ae.context.shuffleIds.asScala.keys
}
```

When `AdaptiveSparkPlanExec` is wrapped by `ColumnarToCarrierRow`, the pattern matching fails, making shuffle IDs inaccessible.

### **Root Cause**

In `GlutenWriterColumnarRules.injectFakeRowAdaptor()`, when the child is an `AdaptiveSparkPlanExec`, the original implementation:
1. Created a new `AdaptiveSparkPlanExec` with `supportsColumnar=true`
2. Wrapped this with `genColumnarToCarrierRow()` → `ColumnarToCarrierRow(AdaptiveSparkPlanExec(...))`

This structure hides `AdaptiveSparkPlanExec` inside `ColumnarToCarrierRow`, breaking any external pattern matching.

### **Solution**

Refactored the wrapping logic to:
1. Wrap `aqe.inputPlan` with `genColumnarToCarrierRow()` first → `ColumnarToCarrierRow(inputPlan)`
2. Create a new `AdaptiveSparkPlanExec` with the wrapped child → `AdaptiveSparkPlanExec(ColumnarToCarrierRow(...))`
3. Set `supportsColumnar=false` since the child is already wrapped

### Gluten version

main branch

### Spark version

spark-4.0.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.