[FEA] category dtype support in parquet writer
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
writing code with `import cudf as pd`
**Describe the solution you'd like**
same behavior as `import pandas as pd`
```
In [1]: import cudf as pd
In [2]: pd.__version__
Out[2]: '22.12.01'
In [3]: df = pd.DataFrame({'a': ['one','two','three'] * 10})
In [4]: df.info()
RangeIndex: 30 entries, 0 to 29
Data columns (total 1 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 a 30 non-null object
dtypes: object(1)
memory usage: 234.0+ bytes
In [5]: df.a = df.astype('category')
In [6]: df.info()
RangeIndex: 30 entries, 0 to 29
Data columns (total 1 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 a 30 non-null category
dtypes: category(1)
memory usage: 57.0 bytes
In [7]: df.to_parquet('df.parquet')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[7], line 1
----> 1 df.to_parquet('df.parquet')
File .../lib/python3.8/site-packages/cudf/core/dataframe.py:6287, in DataFrame.to_parquet(self, path, engine, compression, index, partition_cols, partition_file_name, partition_offsets, statistics, metadata_file_path, int96_timestamps, row_group_size_bytes, row_group_size_rows, max_page_size_bytes, max_page_size_rows, storage_options, return_metadata, *args, **kwargs)
6284 """{docstring}"""
6285 from cudf.io import parquet
-> 6287 return parquet.to_parquet(
6288 self,
6289 path=path,
6290 engine=engine,
6291 compression=compression,
6292 index=index,
6293 partition_cols=partition_cols,
6294 partition_file_name=partition_file_name,
6295 partition_offsets=partition_offsets,
6296 statistics=statistics,
6297 metadata_file_path=metadata_file_path,
6298 int96_timestamps=int96_timestamps,
6299 row_group_size_bytes=row_group_size_bytes,
6300 row_group_size_rows=row_group_size_rows,
6301 max_page_size_bytes=max_page_size_bytes,
6302 max_page_size_rows=max_page_size_rows,
6303 storage_options=storage_options,
6304 return_metadata=return_metadata,
6305 *args,
6306 **kwargs,
6307 )
File .../lib/python3.8/contextlib.py:75, in ContextDecorator.__call__..inner(*args, **kwds)
72 @wraps(func)
73 def inner(*args, **kwds):
74 with self._recreate_cm():
---> 75 return func(*args, **kwds)
File .../lib/python3.8/site-packages/cudf/io/parquet.py:700, in to_parquet(df, path, engine, compression, index, partition_cols, partition_file_name, partition_offsets, statistics, metadata_file_path, int96_timestamps, row_group_size_bytes, row_group_size_rows, max_page_size_bytes, max_page_size_rows, storage_options, return_metadata, *args, **kwargs)
698 if partition_cols is None or col not in partition_cols:
699 if df[col].dtype.name == "category":
--> 700 raise ValueError(
701 "'category' column dtypes are currently not "
702 + "supported by the gpu accelerated parquet writer"
703 )
705 if partition_cols:
706 if metadata_file_path is not None:
ValueError: 'category' column dtypes are currently not supported by the gpu accelerated parquet writer
In [8]: df.to_pandas().to_parquet('df.parquet')
In [9]: %ls df.parquet
df.parquet
```
Contributor guide
Assessment
This issue has not been assessed yet.