[Python] Check `write_table` options are accepted by dataset writer
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Some ParquetWriter options accepted by `pq.write_table` are not registered in `ParquetFileWriteOptions`, so `pq.write_to_dataset` and `ds.write_dataset` reject them:
- `write_time_adjusted_to_utc` #47745
- `bloom_filter_options` #49377 (also missing in `ParquetWriter.__init__`)
```python
pq.write_to_dataset(table, path, bloom_filter_options={'a': True})
```
Output
```console
File "_/python/pyarrow/parquet/core.py", line 2279, in write_to_dataset
write_options = parquet_format.make_write_options(**kwargs)
pyarrow._dataset_parquet.ParquetFileFormat.make_write_options
( opts).update(**kwargs)
File "pyarrow/_dataset_parquet.pyx", line 628, in pyarrow._dataset_parquet.ParquetFileWriteOptions.update
raise TypeError("unexpected parquet write option: " + name)
TypeError: unexpected parquet write option: bloom_filter_options
```
- (? `store_schema` commit 56b90a44d0 )
Similar to `use_content_defined_chunking` getting added in #47499 / #47498
[implementor's note](https://github.com/apache/arrow/blame/51bfc67ef646b5616003c9630e615618fbe48481/python/pyarrow/parquet/core.py#L2022)
A test that every option in `inspect.signature(pq.write_table).parameters` is accepted by `ParquetFileFormat.make_write_options` may be considered to prevent this in the future.
### Component(s)
Python
Contributor guide
Research direction
Start in python/pyarrow/parquet/core.py around the write_to_dataset implementation and the ParquetFileWriteOptions.make_write_options path. Compare options accepted by inspect.signature(pq.write_table) with ParquetFileFormat.make_write_options, then add coverage showing the listed options are accepted and run the relevant Python tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100