NVIDIA / NVIDIA/cudf

[FEA] category dtype support in parquet writer

Open
#12,496 0 comments 0 reactions 0 assignees View on GitHub
0 - Backlog cuIO feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
writing code with `import cudf as pd`

**Describe the solution you'd like**
same behavior as `import pandas as pd`

```
In [1]: import cudf as pd

In [2]: pd.__version__
Out[2]: '22.12.01'

In [3]: df = pd.DataFrame({'a': ['one','two','three'] * 10})

In [4]: df.info()

RangeIndex: 30 entries, 0 to 29
Data columns (total 1 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 a 30 non-null object
dtypes: object(1)
memory usage: 234.0+ bytes

In [5]: df.a = df.astype('category')

In [6]: df.info()

RangeIndex: 30 entries, 0 to 29
Data columns (total 1 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 a 30 non-null category
dtypes: category(1)
memory usage: 57.0 bytes

In [7]: df.to_parquet('df.parquet')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[7], line 1
----> 1 df.to_parquet('df.parquet')

File .../lib/python3.8/site-packages/cudf/core/dataframe.py:6287, in DataFrame.to_parquet(self, path, engine, compression, index, partition_cols, partition_file_name, partition_offsets, statistics, metadata_file_path, int96_timestamps, row_group_size_bytes, row_group_size_rows, max_page_size_bytes, max_page_size_rows, storage_options, return_metadata, *args, **kwargs)
6284 """{docstring}"""
6285 from cudf.io import parquet
-> 6287 return parquet.to_parquet(
6288 self,
6289 path=path,
6290 engine=engine,
6291 compression=compression,
6292 index=index,
6293 partition_cols=partition_cols,
6294 partition_file_name=partition_file_name,
6295 partition_offsets=partition_offsets,
6296 statistics=statistics,
6297 metadata_file_path=metadata_file_path,
6298 int96_timestamps=int96_timestamps,
6299 row_group_size_bytes=row_group_size_bytes,
6300 row_group_size_rows=row_group_size_rows,
6301 max_page_size_bytes=max_page_size_bytes,
6302 max_page_size_rows=max_page_size_rows,
6303 storage_options=storage_options,
6304 return_metadata=return_metadata,
6305 *args,
6306 **kwargs,
6307 )

File .../lib/python3.8/contextlib.py:75, in ContextDecorator.__call__..inner(*args, **kwds)
72 @wraps(func)
73 def inner(*args, **kwds):
74 with self._recreate_cm():
---> 75 return func(*args, **kwds)

File .../lib/python3.8/site-packages/cudf/io/parquet.py:700, in to_parquet(df, path, engine, compression, index, partition_cols, partition_file_name, partition_offsets, statistics, metadata_file_path, int96_timestamps, row_group_size_bytes, row_group_size_rows, max_page_size_bytes, max_page_size_rows, storage_options, return_metadata, *args, **kwargs)
698 if partition_cols is None or col not in partition_cols:
699 if df[col].dtype.name == "category":
--> 700 raise ValueError(
701 "'category' column dtypes are currently not "
702 + "supported by the gpu accelerated parquet writer"
703 )
705 if partition_cols:
706 if metadata_file_path is not None:

ValueError: 'category' column dtypes are currently not supported by the gpu accelerated parquet writer

In [8]: df.to_pandas().to_parquet('df.parquet')

In [9]: %ls df.parquet
df.parquet
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.