apache / apache/arrow

[Python] Check `write_table` options are accepted by dataset writer

Open
#51,220 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Some ParquetWriter options accepted by `pq.write_table` are not registered in `ParquetFileWriteOptions`, so `pq.write_to_dataset` and `ds.write_dataset` reject them:

- `write_time_adjusted_to_utc` #47745
- `bloom_filter_options` #49377 (also missing in `ParquetWriter.__init__`)

```python
pq.write_to_dataset(table, path, bloom_filter_options={'a': True})
```

Output
```console
File "_/python/pyarrow/parquet/core.py", line 2279, in write_to_dataset
write_options = parquet_format.make_write_options(**kwargs)
pyarrow._dataset_parquet.ParquetFileFormat.make_write_options
( opts).update(**kwargs)
File "pyarrow/_dataset_parquet.pyx", line 628, in pyarrow._dataset_parquet.ParquetFileWriteOptions.update
raise TypeError("unexpected parquet write option: " + name)
TypeError: unexpected parquet write option: bloom_filter_options
```
- (? `store_schema` commit 56b90a44d0 )

Similar to `use_content_defined_chunking` getting added in #47499 / #47498

[implementor's note](https://github.com/apache/arrow/blame/51bfc67ef646b5616003c9630e615618fbe48481/python/pyarrow/parquet/core.py#L2022)

A test that every option in `inspect.signature(pq.write_table).parameters` is accepted by `ParquetFileFormat.make_write_options` may be considered to prevent this in the future.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start in python/pyarrow/parquet/core.py around the write_to_dataset implementation and the ParquetFileWriteOptions.make_write_options path. Compare options accepted by inspect.signature(pq.write_table) with ParquetFileFormat.make_write_options, then add coverage showing the listed options are accepted and run the relevant Python tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.