VeloxRssSortShuffleReaderDeserializer: infinite loop on EOS mid-page and corrupted deserialization on multi-window Presto pages
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
### Description
We hit two bugs in the rss_sort shuffle reader (`VeloxRssSortShuffleReaderDeserializer::VeloxInputStream`) in our production environment with Celeborn (rss_sort + hash partitioning). Both reproduce on current main — regression tests submitted in #12983 fail.
### Bug 1: infinite loop on EOS mid-page
`GlutenByteInputStream::readBytes()` drives `next(true)` in a `for(;;)` loop. When a page header declares more bytes than the stream actually holds (truncated partition data), the reader hits EOS mid-page — but `VeloxInputStream::next()` ignores its `throwIfPastEnd` argument and silently returns on EOS, so the loop never exits.
Symptom in production: the task appears stuck (thread in RUNNABLE, no progress, no error surfaced).
CI evidence from #12983:
```
C++ exception with description "possible infinite loop: Read() returned 0 for 100 consecutive calls" thrown in the test body.
```
(the test's fake stream cuts the loop short after 100 consecutive EOS reads instead of hanging)
Additionally, `next()` stores the `Read()` result into an unsigned `offset_` without a signed guard, so a negative return from a buggy upstream corrupts `setRange` instead of failing fast.
### Bug 2: corrupted deserialization when a page spans multiple read windows
`VeloxInputStream` is a single-window refill stream: each `next()` overwrites the sole `ByteRange` with serde assumes stable multi-range data — for nested types it does `tellp()` → pre-scan the page →`seekp()` back, and it scans the payload via `nextView()` for checksum verification. When an uncompressed page (nested structs) spans multiple ~1MB read windows, the serde's backward seek reads window data already overwritten by a
refill, corrupting the stream.
CI evidence from #12983:
```
Exception: VeloxRuntimeError
Reason: (-673841146 vs. 0) Invalid serialized string size: -673841146
Function: readLengthPrefixedString
# 3 ...readLengthPrefixedString(ByteInputStream*)
# 6 ...PrestoVectorSerde::deserialize(...)
# 9 gluten::VeloxRssSortShuffleReaderDeserializer::next()
```
### Impact
- Bug 1: task hangs forever with no error — requires external timeout to kill, hard to diagnose from logs.
- Bug 2: wrong data / spurious deserialization errors for pages larger than one read window (~1MB) containing nested types.
### Fix
We have a fix ready and will submit a PR shortly (referencing this issue).
### Gluten version
main branch, Gluten-1.3
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
Contributor guide
Research direction
Start with VeloxRssSortShuffleReaderDeserializer::VeloxInputStream and the regression tests referenced in #12983. Trace next(), readBytes(), and nextView() through EOS handling and multi-window refills. Done means truncated pages fail promptly and pages spanning read windows deserialize without corruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100