NVIDIA / NVIDIA/cudf

[BUG] .to_parquet() and .to_csv() fails and get OOM with large DataFrames.

Open
#10,426 7 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf Performance Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Writting large DataFrames to disk fails and gets memory error:
>>>df.to_parquet('myfile.parquet')
>>>RuntimeError: CUDA error at: /home/giba/anaconda3/envs/rapids-22.02/include/rmm/cuda_stream_view.hpp:81: cudaErrorIllegalAddress an illegal memory access was encountered

**Steps/Code to reproduce bug**
My DataFrame have shape: (414395052, 4)
dtypes: var0 int32, var1 int32, var2 int8, var3 int8
df.memory_usage().sum() returns: 4143950520 (4GB)

**Expected behavior**
File write to disk in .parquet or .csv format without issues.

**Environment overview (please complete the following information)**
Used conda with default RAPIDS 22.02 install.

**Environment details**
Using a 32GB V100 GPU.

**Additional context**
Sending cudf Dataframe to Pandas then calling .to_parquet() works:
>>> df.to_pandas().to_parquet('myfile.parquet')

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.