[VL] Gluten TableScan's metric(raw input bytes) is larger than Vanilla's
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
The gluten TableScan operator's metric(raw input bytes) is larger than the vanilla spark, it's curious and abnormal, upon checking the code, I found the following info.
First, this metric comes from `ReadFileInputStream::read()` and is stored in `IoStatistics.rawBytesRead_`, it means how much data is read from the file with the calling of `readFile_->pread()`.
the class `BufferedInput`, which wraps `ReadFileInputStream`, buffers the file's data in memory to enhance reading performance, and it is used by the class `ReaderBase` and `StripeReaderBase`.
`filePreloadThreshold` has a default setting of `8M`, and the preload will occur in the constructor of `ReaderBase`.
A local file's size may be 10M, but the same segment may be read multiple times, resulting in `rawBytesRead_` being greater than the file's actual size.
**What should we do about this problem?**
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Trace ReadFileInputStream::read() through IoStatistics.rawBytesRead_ and the readFile_->pread() call, then inspect how BufferedInput, ReaderBase, StripeReaderBase, and filePreloadThreshold affect repeated or preloaded reads. Compare the resulting accounting with the vanilla TableScan metric and establish a verified definition for what the raw input bytes metric should report.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100