NVIDIA / NVIDIA/cudf

[BUG] `uses_custom_row_bounds` should not be hardcoded to true in `chunked_parquet_reader`

Open
#15,779 0 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

Related to #15764

Changing `uses_custom_row_bounds` from hardcoded `true` to the below (correct) logic for `chunked_parquet_reader` tests from `ParquetChunkedReaderTest.TestChunkedReadWithListsNoNulls` start failing (see the log below). This happens if `uses_custom_row_bounds = false` and eventually [ComputePageSizes](https://github.com/rapidsai/cudf/blob/branch-24.06/cpp/src/io/parquet/reader_impl_preprocess.cu#L1473) is not computed.

```c++
bool const uses_custom_row_bounds = options.get_num_rows().has_value() or options.get_skip_rows() != 0;
```

This has been done to avoid a couple other bugs as described by @nvdbaranec and @ttnghia:

> We are calling `read_chunk_internal()` in a few places where we're only passing`read_mode`. But if chunked_read_size is > 0 we need to force that to be true. If we don't, various important things don't happen for all the remaining chunks. Basically if this code doesn't execute, it'll be bad

https://github.com/rapidsai/cudf/blob/bd7b5101cfb77b5b0210cc1d7b78a587821b9bf6/cpp/src/io/parquet/reader_preprocess.cu#L499

Test fail log:
```bash
/home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 150000
31: rhs_size
31: Which is: 750000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:624: <-- line of failure
31:
31: unknown file: Failure
31: C++ exception with description "CUDF failure at: /home/coder/cudf/cpp/include/cudf/detail/null_mask.cuh:380: End index cannot be smaller than the starting index." thrown in the test body.
31: [ FAILED ] ParquetChunkedReaderTest.TestChunkedReadWithListsNoNulls (49 ms)
31: [ RUN ] ParquetChunkedReaderTest.TestChunkedReadWithListsHavingNulls
31: unknown file: Failure
31: C++ exception with description "CUDF failure at:/home/coder/cudf/cpp/src/io/parquet/reader_impl.cpp:328: Parquet data decode failed with code(s) 0x77777777" thrown in the test body.
31: [ FAILED ] ParquetChunkedReaderTest.TestChunkedReadWithListsHavingNulls (16 ms)
31: [ RUN ] ParquetChunkedReaderTest.TestChunkedReadWithStructsOfLists
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 150000
31: rhs_size
31: Which is: 1500000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:804: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 100000
31: rhs_size
31: Which is: 500000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:809: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:562: Failure
31: Failed
31: depth 2
31: first difference: lhs[0] = -1145324613, rhs[0] = 0
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:821: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 150000
31: rhs_size
31: Which is: 1050000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:834: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 150000
31: rhs_size
31: Which is: 600000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:840: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 150000
31: rhs_size
31: Which is: 600000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:846: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 150000
31: rhs_size
31: Which is: 300000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:852: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 100000
31: rhs_size
31: Which is: 500000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:858: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 100000
31: rhs_size
31: Which is: 500000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:864: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 100000
31: rhs_size
31: Which is: 300000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:870: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:562: Failure
31: Failed
31: depth 1
31: first difference: lhs[9612] = 9612++��\0��\0��\0��\0��\0��, rhs[9612] = 9612++++++++++++++++++++9612
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:876: <-- line of failure
31:
31: [ FAILED ] ParquetChunkedReaderTest.TestChunkedReadWithStructsOfLists (609 ms)
31: [ RUN ] ParquetChunkedReaderTest.TestChunkedReadWithListsOfStructs
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:562: Failure
31: Failed
31: first difference: lhs[0] = 0, rhs[0] = 94272
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:945: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:258: Failure
31: Expected equality of these values:
31: lhs_size
31: Which is: 150000
31: rhs_size
31: Which is: 1500000
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:945: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:262: Failure
31: Expected equality of these values:
31: lhs.null_count()
31: Which is: 25000
31: rhs.null_count()
31: Which is: 20427
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:950: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:562: Failure
31: Failed
31: depth 2
31: first difference: lhs[9696] = 30000, rhs[9696] = 9696
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:957: <-- line of failure
31:
31: /home/coder/cudf/cpp/tests/utilities/column_utilities.cu:562: Failure
31: Failed
31: depth 2
31: first difference: lhs[479] = 7`�\0\0`�\0\0, rhs[479] = 719719719
31: Google Test trace:
31: /home/coder/cudf/cpp/tests/io/parquet_chunked_reader_test.cu:962: <-- line of failure
31:
31: CMake Error at /home/coder/cudf/cpp/build/latest/rapids-cmake/run_gpu_test.cmake:34 (execute_process):
31: execute_process failed command indexes:
31:
31: 1: "Abnormal exit with child return code: Segmentation fault"
```

**Steps/Code to reproduce bug**
Modify in [reader_impl.hpp](https://github.com/rapidsai/cudf/blob/branch-24.06/cpp/src/io/parquet/reader_impl.hpp)
```c++
[[nodiscard]] bool uses_custom_row_bounds(read_mode mode) const
{
return (mode == read_mode::READ_ALL)
? (_options.num_rows.has_value() or _options.skip_rows != 0)
: true;
}
```
to
```c++
[[nodiscard]] bool uses_custom_row_bounds(read_mode mode) const
{
return _options.num_rows.has_value() or _options.skip_rows != 0;
}
```

**Expected behavior**
The gtests should not fail.

**Environment overview**
- Environment location: Bare-metal (dgx-05)
- Method of cuDF install: cuDF devcontainer (conda, cuda 12.2)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.