`read_csv` with `include_path_column` and `dtype_backend='pyarrow'` generates a mix of String and Categorical which hurts Parquet usage
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
Using `read_csv` to read CSVs as `pyarrow` dtypes with `dtype_backend='pyarrow'`, then outputting a Parquet file set gives us a mix of PyArrow and `categorical` outputs. The categorical is the added `path` if `include_path_column=True`.
Perhaps if `dtype_backend='pyarrow'` the behaviour of `include_path_column=True` should use a PyArrow string, not a `categorical`?
This came up when trying to load my Parquet output from Dask into Polars: https://github.com/pola-rs/polars/issues/8925
By subsequently removing `include_path_column=True` from the `dd.read_csv`, the pipeline generates an all-PyArrow output with no categorical (i.e. no `path` column but the 14 expected columns are all present).
Having removed `include_path_column` from this Parquet output, the resulting Parquet file-set can be read with Polars with no need to sub-select away the "path" column.
**Partial example**:
```python
# define a function to read each csv file
def read_results(file, sep):
df = dd.read_csv(file,
delimiter=sep,
doublequote=False,
on_bad_lines='warn',
include_path_column=True, # FORCES a categorical column
parse_dates=['test_date'],
dtype_backend='pyarrow', # REQUEST all pyarrow dtypes and no categoricals
)
return df
# some lines ommitted for date parsing and binding read_results to previously sniffed varying csv formats...
# inferred dtypes are all pyarrow as expected
ddf.dtypes
test_id int64[pyarrow]
test_date datetime64[ns]
test_class_id int64[pyarrow]
fuel_type string[pyarrow]
...14 in total, clipped for brevity, all pyarrow types
dtype: object
# generate the parquet output, let it process for 10 minutes
f = ddf.to_parquet('test_result.parquet', write_index=False, overwrite=True, compute=False).persist()
progress(f)
# now roundtrip the result back in and check what we've got
ddf_result = dd.read_parquet('test_result.parquet')
ddf_result.info(verbose=True)
Index: 639506962 entries, 0 to 791798
Data columns (total 15 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 test_id 639506962 non-null int64[pyarrow]
2 test_date 639506962 non-null datetime64[ns]
3 test_class_id 639506962 non-null int64[pyarrow]
11 fuel_type 639506962 non-null string
...14 in total as before and all pyarrow until...
14 path 639506962 non-null category # UNEXPECTED non-arrow category
dtypes: category(1), datetime64[ns](2), int64[pyarrow](5), string(7)
memory usage: 65.2 GB
```
**Anything else we need to know?**:
I'm guessing this is old-and-sensible behaviour from when NumPy was the default in Pandas but now that Arrow is a first-class citizen, maybe this behaviour needs updating?
**Environment**:
- Dask version: '2023.5.0'
- Python version: 3.11
- Operating System: Linux Mint, Ubuntu
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.