apache / apache/gluten

[VL] Gluten TableScan's metric(raw input bytes) is larger than Vanilla's

Open
#3,292 6 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

The gluten TableScan operator's metric(raw input bytes) is larger than the vanilla spark, it's curious and abnormal, upon checking the code, I found the following info.

First, this metric comes from `ReadFileInputStream::read()` and is stored in `IoStatistics.rawBytesRead_`, it means how much data is read from the file with the calling of `readFile_->pread()`.

the class `BufferedInput`, which wraps `ReadFileInputStream`, buffers the file's data in memory to enhance reading performance, and it is used by the class `ReaderBase` and `StripeReaderBase`.

`filePreloadThreshold` has a default setting of `8M`, and the preload will occur in the constructor of `ReaderBase`.

A local file's size may be 10M, but the same segment may be read multiple times, resulting in `rawBytesRead_` being greater than the file's actual size.

**What should we do about this problem?**

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Trace ReadFileInputStream::read() through IoStatistics.rawBytesRead_ and the readFile_->pread() call, then inspect how BufferedInput, ReaderBase, StripeReaderBase, and filePreloadThreshold affect repeated or preloaded reads. Compare the resulting accounting with the vanilla TableScan metric and establish a verified definition for what the raw input bytes metric should report.

Written by the indexing model from the issue text.

Assessment

Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.