apache / apache/gluten

[CORE] Incorrect wholestage id in WholeStageTransformerExec

Open
#11,251 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

In vanilla spark, the stage id of the `WholeStageCodegenExec` always starts with 1 in each SQL query
Image

However in Gluten, the stage id is generated from a global counter, which accumulates across different queries. This number can grow very large in a long running spark session.
Image

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing WholeStageTransformerExec and comparing its stage-id handling with WholeStageCodegenExec, the two execution classes named in the report. Reproduce the behavior across multiple SQL queries and identify how the counter is scoped; done means stage IDs follow the per-query behavior described and regression coverage verifies it.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.