NVIDIA / NVIDIA/cudf

[BUG] cudaErrorIllegalAddress encounted when reading a parquet file

Open
#19,642 9 comments 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Reading a parquet file causes error 'cudaErrorIllegalAddress an illegal memory access was encountered'

**Steps/Code to reproduce bug**
```python
import cudf
cudf.read_parquet("example.parquet")
```
generates error message
```
Traceback (most recent call last):
File "", line 1, in
File "/home/qing/.local/lib/python3.12/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/qing/.local/lib/python3.12/site-packages/cudf/io/parquet.py", line 922, in read_parquet
df = _parquet_to_frame(
^^^^^^^^^^^^^^^^^^
File "/home/qing/.local/lib/python3.12/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/qing/.local/lib/python3.12/site-packages/cudf/io/parquet.py", line 1070, in _parquet_to_frame
return _read_parquet(
^^^^^^^^^^^^^^
File "/home/qing/.local/lib/python3.12/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/qing/.local/lib/python3.12/site-packages/cudf/io/parquet.py", line 1282, in _read_parquet
tbl_w_meta = plc.io.parquet.read_parquet(options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pylibcudf/io/parquet.pyx", line 319, in pylibcudf.io.parquet.read_parquet
File "pylibcudf/io/parquet.pyx", line 337, in pylibcudf.io.parquet.read_parquet
MemoryError: std::bad_alloc: CUDA error (failed to allocate 176 bytes) at: /__w/rmm/rmm/cpp/include/rmm/mr/device/cuda_memory_resource.hpp:62: cudaErrorIllegalAddress an illegal memory access was encountered
```
It looks like the error comes from
https://github.com/rapidsai/cudf/blob/branch-25.08/python/pylibcudf/pylibcudf/io/parquet.pyx#L319-L337

Actually, we have a dataset of parquet files like this one, but only a very few of them (< 1%) cause crash.

**Expected behavior**
Successful reading. The file can be parsed without any error by pyarrow
```python
import pyarrow.parquet as pq
pq.read_table("example.parquet")
```
outputs
```
pyarrow.Table
Timestamp: list
child 0, element: float
----
Timestamp: [[[19850,3,0,0.190126],[19850,3,0,0.487079],...,[19850,15,1796,0.674507],[19850,15,1799,0.626622]]]
```

**Environment overview (please complete the following information)**
```
- Environment location: [local]
- Method of cuDF install: [pip]
`pylibcudf-cu12 25.8.0`
```
**Environment details**
```
***OS Information***
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
Linux s01 6.8.0-64-generic #67-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 15 20:23:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

***GPU Information***
NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9
NVIDIA GeForce RTX 4080 Super
```

Any help will be appreciated.

[example.parquet.zip](https://github.com/user-attachments/files/21700064/example.parquet.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.