apache / apache/arrow

[Python] `ds.write_dataset` doesn't allow feather compression

Open
#33,537 0 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

`ds.write_dataset` allows specifying Parquet compression, for example:
```python

import pandas as pd
import pyarrow as pa
import pyarrow.dataset as ds

df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})

df = pa.Table.from_pandas(df)

ds.write_dataset(
    df,
    base_dir='test',
    format='parquet',
  file_options=ds.ParquetFileFormat().make_write_options(compression='snappy'))
```
However, such trick (the `file_options` argument) doesn't work for feather, as the following code gives me an error:
```python

import pandas as pd
import pyarrow as pa
import pyarrow.dataset as ds

df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})

df = pa.Table.from_pandas(df)

ds.write_dataset(
    df,
    base_dir='test',
    format='feather',

    file_options=ds.FeatherFileFormat().make_write_options(compression='uncompressed'))
```
The error: `TypeError: FeatherFileFormat.make_write_options() takes no keyword arguments`

**Environment**: Ubuntu 22.04
**Reporter**: [Yu Zhu](https://issues.apache.org/jira/browse/ARROW-18370)

**Note**: *This issue was originally created as [ARROW-18370](https://issues.apache.org/jira/browse/ARROW-18370). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Python ds.write_dataset example with format='feather' and inspect FeatherFileFormat.make_write_options, which is the entry point named in the report. Trace how file_options are handled for Feather compared with Parquet. Done means Feather dataset writes accept the requested compression option without the reported TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.