NVIDIA / NVIDIA/cudf

[BUG] ValueError: 'category' column dtypes are currently not supported by the gpu accelerated parquet writer

Open
#10,115 5 comments 0 reactions 0 assignees View on GitHub
cuIO feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

The following exception is thrown when a column of type 'int' is written using to_parquet, and read with read_parquet and re-written with to-parquet again.

So, the first to_parquet is changing the `int` col type to catg, but when reading again, and the same file is re-written it fails.

Exception: ValueError: 'category' column dtypes are currently not supported by the gpu accelerated parquet writer

Reproducer Code:
```
import cudf
df = cudf.DataFrame()

df['a'] = [0 , 0 , 0, 1]
df['b'] = [0 , 0 , 0, 1]

df['a'] = df.a.astype('int')
df.to_parquet('test', partition_cols=['a'], partition_file_name=f'0.parquet')
df = cudf.read_parquet('test/a=0/0.parquet')
df.to_parquet('test.pq')
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.