NVIDIA / NVIDIA/cudf

[BUG] Debug builds crash with memcpy: source range is invalid in preprocess_levels_kernel (RLE level-stream shared-memory staging)

Open
#23,662 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

In **Debug** builds only, several parquet reader tests crash with a device-side assert:
```
libcudacxx/include/cuda/std/__cstring/memcpy.h:49: memcpy: block: [N,0,0], thread: [M,0,0]
Assertion memcpy: source range is invalid failed.
CUDA Error detected. cudaErrorAssert device-side assert triggered
```
The crash happens in `preprocess_levels_kernel` (`cpp/src/io/parquet/decode_preprocess.cu`), which decodes definition/repetition levels via `rle_stream::decode_next_chunked()` in `cpp/src/io/parquet/rle_stream.cuh`.

Bisected this to the shared-memory staging path added in `rle_stream::init()`:

```cpp
if (_smem_stage != nullptr) {
auto* const smem_stage = static_cast(cuda::std::assume_aligned<16>(_smem_stage));
auto const len = static_cast(cuda::std::distance(_start, _end));
if (len > 0 && len <= stage_capacity) {
cuda::memcpy_async(group, _smem_stage, _start, static_cast(len), *_copy_barrier);
cur = smem_stage;
end = smem_stage + len;
}
}
```

Passing `nullptr` instead of `stage` for both the REPETITION and DEFINITION decoder `init()` calls in
`decode_preprocess.cu` (i.e. disabling shared-memory staging entirely and always decoding from global memory) makes all failing tests pass. Re-enabling staging for either decoder reproduces the crash. I was not able to pin the exact defective line further — `cuda-gdb` crashes with its own internal error.
Using `compute-sanitizer --show-backtrace=yes` shows host-side stacks and SASS/line-table correlation which may help.

Bisected to the combination of:
- #23090
- #23271

Both touch `cpp/src/io/parquet/rle_stream.cuh` / `cpp/src/io/parquet/decode_preprocess.cu`.

**Steps/Code to reproduce bug**

Build `libcudf` in **Debug** mode (`CMAKE_BUILD_TYPE=Debug`), then run any of:

```
UTILITIES_TEST --gtest_filter=PinnedMemoryTest.MemoryResourceGetAndSet
PARQUET_TEST --gtest_filter=ParquetChunkedReaderTest.TestChunkedReadSimpleData
HYBRID_SCAN_TEST --gtest_filter=HybridScanMultifileTest.MaterializeLists
LARGE_STRINGS_TEST --gtest_filter=ParquetStringsTest.ChunkedReadNestedLargeStrings
STREAM_IO_PARQUET_TEST --gtest_filter=ParquetTest.ChunkedOperations
```

All five fail identically with the assert above, and all five pass when shared-memory staging is disabled per the bisection described.

Contributor guide

Open the contributing guide

Research direction

Start with cpp/src/io/parquet/rle_stream.cuh and cpp/src/io/parquet/decode_preprocess.cu, focusing on the shared-memory staging path in rle_stream::init() and the REPETITION and DEFINITION decoder init() calls. Build libcudf in Debug mode and run a listed failing filter, using compute-sanitizer --show-backtrace=yes as suggested. Done means the staging path no longer triggers the invalid memcpy assertion and the affected parquet tests pass with staging enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.