apache / apache/gluten

[VL] HeuristicTransform hide all rules transform details

Open
#9,311 6 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

in spark, I can use spark.sql.planChangeLog.level to check the details of plan transformation, and it works in gluten before, but now, all I see is HeuristicTransform.

@zhztheplayer would you help unwrap the transformation loop to make spark.sql.planChangeLog.level works as before?
btw, spark.gluten.sql.transform.logLevel does not work for HeuristicTransform.

```
=== Applying Rule org.apache.gluten.extension.columnar.heuristic.HeuristicTransform took 53 ms ===
Execute InsertIntoHadoopFsRelationCommand file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest, false, [dt#71], 4 buckets, bucket columns: [bid], Parquet, [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC], Overwrite, `spark_catalog`.`default`.`dest`, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, org.apache.spark.sql.execution.datasources.CatalogFileIndex(file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest), [id, bid, dt] Execute InsertIntoHadoopFsRelationCommand file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest, false, [dt#71], 4 buckets, bucket columns: [bid], Parquet, [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC], Overwrite, `spark_catalog`.`default`.`dest`, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, org.apache.spark.sql.execution.datasources.CatalogFileIndex(file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest), [id, bid, dt]
!+- WriteFiles +- VeloxColumnarWriteFiles Parquet, [dt#71], 4 buckets, bucket columns: [bid], [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC]
! +- Sort [dt#71 ASC NULLS FIRST, pmod((hive-hash(bid#63) & 2147483647), 4) ASC NULLS FIRST], false, 0 :- WriteFilesExecTransformer Parquet, [dt#71], 4 buckets, bucket columns: [bid], [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC]
! +- Project [id#61, bid#63, empty2null(dt#62) AS dt#71] : +- ProjectExecTransformer [id#61, bid#63, dt#71]
! +- FileScan parquet spark_catalog.default.src[id#61,dt#62,bid#63] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.G..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct : +- SortExecTransformer [dt#71 ASC NULLS FIRST, _pre_0#72 ASC NULLS FIRST], false, 0, 0
! : +- Project [id#61, bid#63, dt#71, pmod((hive-hash(bid#63) & 2147483647), 4) AS _pre_0#72]
! : +- ProjectExecTransformer [id#61, bid#63, empty2null(dt#62) AS dt#71]
! : +- FileScanTransformer parquet spark_catalog.default.src[id#61,dt#62,bid#63] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.G..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct NativeFilters: []
! +- !WriteFiles
!
```

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the HeuristicTransform rule and the Spark plan-change logging path, focusing on why spark.sql.planChangeLog.level only reports the wrapper. Compare that behavior with spark.gluten.sql.transform.logLevel and verify that completed transformations expose their individual rule details as before.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.