Gluten UI display raw input bytes instead of input bytes for batch scan
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
We're trying to compare the performance between gluten and vanilla spark for iceberg datasource, and we found that for the same table scan stage, the input bytes shown a large difference. We wonder if it's the gluten improvement of filter pushdown, but we observed that the metrics for BatchScan shown that there were 2 type of input bytes of gluten: raw input bytes (which we understand as the raw input from disk, without decompressing and decoding) and input bytes total (which is the input vector size after decompressing and decoding)
While for vanilla spark, it shows the input size which is comparable with the second size (input bytes total) of gluten, but gluten display the first one in the front of stage ui. Could we unify this with vanilla spark to avoid potential misunderstanding of users?
Contributor guide
Assessment
This issue has not been assessed yet.