NVIDIA / NVIDIA/cudf

[BUG] Varint decoders wrap overflowing values and shift out of range

Open
#23,478 0 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Several of cudf's varint decoders have overflow issue: when performing a shift operation, a terminating group with payload exceeding the remaining bits of the target type will silently truncated and wrap (overflow). As such, malformed ORC/Avro/Parquet metadata with such over long input value will decode to a wrong-but-plausible value rather than failing cleanly.

For example, `protobuf_reader::get` in `cpp/src/io/orc/orc.hpp:414`:

```cpp
uint64_t v = 0;
for (uint32_t l = 0; l < sizeof(v) * 8; l += 7) { // guards the shift count only
uint64_t c = get();
v |= (c & 0x7f) << l; // at l == 63 any value above 1 overflows
if (c < 0x80) return v;
}
CUDF_FAIL("Invalid varint: exceeds maximum encoded length"); // only fires past 10 bytes
```

For a 10-byte input `{0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x02}`, the returns value will be `0` for an out-of-range varint which is incorrect.

Affected decoders — host-side:

- `cpp/src/io/orc/orc.hpp:414` — `protobuf_reader::get`, shown above (`CUDF_FAIL` fires only when no group terminates within 10 bytes).
- `cpp/src/io/avro/avro.cpp:15` — `container::get_encoded` has the same `len < 64`-only guard and no terminal error at all.

Device-side:

- `cpp/src/io/parquet/page_hdr.cu:79` — `get_u32` has no shift guard at all (`l += 7` is unbounded), so a malformed page header can evaluate `(c & 0x7f) << l` with `l >= 32`, an undefined shift.
- `cpp/src/io/parquet/delta_binary.cuh`, `cpp/src/io/orc/stripe_data.cu`, `cpp/src/io/parquet/rle_stream.cuh`, and `cpp/src/io/avro/avro_gpu.cu` share the same bounded-but-truncating pattern and are worth auditing.

Contributor guide

Open the contributing guide

Research direction

Start with the host-side loops in cpp/src/io/orc/orc.hpp and cpp/src/io/avro/avro.cpp, then compare them with the device-side implementations in cpp/src/io/parquet/page_hdr.cu, delta_binary.cuh, stripe_data.cu, rle_stream.cuh, and avro_gpu.cu. Check the supplied 10-byte example and audit the shared patterns, including the unbounded shift in page_hdr.cu. Done means malformed values exceeding the target width are rejected consistently without changing valid decoding behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.