[Python] `ds.write_dataset` doesn't allow feather compression
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
`ds.write_dataset` allows specifying Parquet compression, for example:
```python
import pandas as pd
import pyarrow as pa
import pyarrow.dataset as ds
df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
df = pa.Table.from_pandas(df)
ds.write_dataset(
df,
base_dir='test',
format='parquet',
file_options=ds.ParquetFileFormat().make_write_options(compression='snappy'))
```
However, such trick (the `file_options` argument) doesn't work for feather, as the following code gives me an error:
```python
import pandas as pd
import pyarrow as pa
import pyarrow.dataset as ds
df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
df = pa.Table.from_pandas(df)
ds.write_dataset(
df,
base_dir='test',
format='feather',
file_options=ds.FeatherFileFormat().make_write_options(compression='uncompressed'))
```
The error: `TypeError: FeatherFileFormat.make_write_options() takes no keyword arguments`
**Environment**: Ubuntu 22.04
**Reporter**: [Yu Zhu](https://issues.apache.org/jira/browse/ARROW-18370)
**Note**: *This issue was originally created as [ARROW-18370](https://issues.apache.org/jira/browse/ARROW-18370). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the Python ds.write_dataset example with format='feather' and inspect FeatherFileFormat.make_write_options, which is the entry point named in the report. Trace how file_options are handled for Feather compared with Parquet. Done means Feather dataset writes accept the requested compression option without the reported TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100