apache / apache/gluten

Gluten UI display raw input bytes instead of input bytes for batch scan

Open
#5,017 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

We're trying to compare the performance between gluten and vanilla spark for iceberg datasource, and we found that for the same table scan stage, the input bytes shown a large difference. We wonder if it's the gluten improvement of filter pushdown, but we observed that the metrics for BatchScan shown that there were 2 type of input bytes of gluten: raw input bytes (which we understand as the raw input from disk, without decompressing and decoding) and input bytes total (which is the input vector size after decompressing and decoding)

While for vanilla spark, it shows the input size which is comparable with the second size (input bytes total) of gluten, but gluten display the first one in the front of stage ui. Could we unify this with vanilla spark to avoid potential misunderstanding of users?

image1
image2

image3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.