[BUG] Varint decoders wrap overflowing values and shift out of range
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Several of cudf's varint decoders have overflow issue: when performing a shift operation, a terminating group with payload exceeding the remaining bits of the target type will silently truncated and wrap (overflow). As such, malformed ORC/Avro/Parquet metadata with such over long input value will decode to a wrong-but-plausible value rather than failing cleanly.
For example, `protobuf_reader::get` in `cpp/src/io/orc/orc.hpp:414`:
```cpp
uint64_t v = 0;
for (uint32_t l = 0; l < sizeof(v) * 8; l += 7) { // guards the shift count only
uint64_t c = get();
v |= (c & 0x7f) << l; // at l == 63 any value above 1 overflows
if (c < 0x80) return v;
}
CUDF_FAIL("Invalid varint: exceeds maximum encoded length"); // only fires past 10 bytes
```
For a 10-byte input `{0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x02}`, the returns value will be `0` for an out-of-range varint which is incorrect.
Affected decoders — host-side:
- `cpp/src/io/orc/orc.hpp:414` — `protobuf_reader::get`, shown above (`CUDF_FAIL` fires only when no group terminates within 10 bytes).
- `cpp/src/io/avro/avro.cpp:15` — `container::get_encoded` has the same `len < 64`-only guard and no terminal error at all.
Device-side:
- `cpp/src/io/parquet/page_hdr.cu:79` — `get_u32` has no shift guard at all (`l += 7` is unbounded), so a malformed page header can evaluate `(c & 0x7f) << l` with `l >= 32`, an undefined shift.
- `cpp/src/io/parquet/delta_binary.cuh`, `cpp/src/io/orc/stripe_data.cu`, `cpp/src/io/parquet/rle_stream.cuh`, and `cpp/src/io/avro/avro_gpu.cu` share the same bounded-but-truncating pattern and are worth auditing.
Contributor guide
Research direction
Start with the host-side loops in cpp/src/io/orc/orc.hpp and cpp/src/io/avro/avro.cpp, then compare them with the device-side implementations in cpp/src/io/parquet/page_hdr.cu, delta_binary.cuh, stripe_data.cu, rle_stream.cuh, and avro_gpu.cu. Check the supplied 10-byte example and audit the shared patterns, including the unbounded shift in page_hdr.cu. Done means malformed values exceeding the target width are rejected consistently without changing valid decoding behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100