[VL] HeuristicTransform hide all rules transform details
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
in spark, I can use spark.sql.planChangeLog.level to check the details of plan transformation, and it works in gluten before, but now, all I see is HeuristicTransform.
@zhztheplayer would you help unwrap the transformation loop to make spark.sql.planChangeLog.level works as before?
btw, spark.gluten.sql.transform.logLevel does not work for HeuristicTransform.
```
=== Applying Rule org.apache.gluten.extension.columnar.heuristic.HeuristicTransform took 53 ms ===
Execute InsertIntoHadoopFsRelationCommand file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest, false, [dt#71], 4 buckets, bucket columns: [bid], Parquet, [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC], Overwrite, `spark_catalog`.`default`.`dest`, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, org.apache.spark.sql.execution.datasources.CatalogFileIndex(file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest), [id, bid, dt] Execute InsertIntoHadoopFsRelationCommand file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest, false, [dt#71], 4 buckets, bucket columns: [bid], Parquet, [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC], Overwrite, `spark_catalog`.`default`.`dest`, org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, org.apache.spark.sql.execution.datasources.CatalogFileIndex(file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.GlutenHiveUDFSuite/dest), [id, bid, dt]
!+- WriteFiles +- VeloxColumnarWriteFiles Parquet, [dt#71], 4 buckets, bucket columns: [bid], [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC]
! +- Sort [dt#71 ASC NULLS FIRST, pmod((hive-hash(bid#63) & 2147483647), 4) ASC NULLS FIRST], false, 0 :- WriteFilesExecTransformer Parquet, [dt#71], 4 buckets, bucket columns: [bid], [serialization.format=1, mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true, partitionOverwriteMode=DYNAMIC]
! +- Project [id#61, bid#63, empty2null(dt#62) AS dt#71] : +- ProjectExecTransformer [id#61, bid#63, dt#71]
! +- FileScan parquet spark_catalog.default.src[id#61,dt#62,bid#63] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.G..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct : +- SortExecTransformer [dt#71 ASC NULLS FIRST, _pre_0#72 ASC NULLS FIRST], false, 0, 0
! : +- Project [id#61, bid#63, dt#71, pmod((hive-hash(bid#63) & 2147483647), 4) AS _pre_0#72]
! : +- ProjectExecTransformer [id#61, bid#63, empty2null(dt#62) AS dt#71]
! : +- FileScanTransformer parquet spark_catalog.default.src[id#61,dt#62,bid#63] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/root/emr-gluten/spark-warehouse/org.apache.spark.sql.execution.G..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct NativeFilters: []
! +- !WriteFiles
!
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by tracing the HeuristicTransform rule and the Spark plan-change logging path, focusing on why spark.sql.planChangeLog.level only reports the wrapper. Compare that behavior with spark.gluten.sql.transform.logLevel and verify that completed transformations expose their individual rule details as before.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100