[R] write_parquet expose similar options as python parquet.write_table ?
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
I've found R write_parquet
```R
write_parquet(
x,
sink,
chunk_size = NULL,
version = "2.4",
compression = default_parquet_compression(),
compression_level = NULL,
use_dictionary = NULL,
write_statistics = NULL,
data_page_size = NULL,
use_deprecated_int96_timestamps = FALSE,
coerce_timestamps = NULL,
allow_truncated_timestamps = FALSE
)
```
missed some options compared to python parquet.write_table
```python
pyarrow.parquet.write_table(table, where, row_group_size=None, version='2.4', use_dictionary=True, compression='snappy', write_statistics=True, use_deprecated_int96_timestamps=None, coerce_timestamps=None, allow_truncated_timestamps=False, data_page_size=None, flavor=None, filesystem=None, compression_level=None, use_byte_stream_split=False, column_encoding=None, data_page_version='1.0', use_compliant_nested_type=False, encryption_properties=None, write_batch_size=None, dictionary_pagesize_limit=None, store_schema=True, **kwargs)
```
maybe R interface should also expose similar options? the most important options to me is
use_byte_stream_split/ column_encoding, since I'm using R to save parquet file, but it can't specify best column encoding for every column and leads bigger file, I have to use pyarrow to convert it again, which is not very ideal.
### Component(s)
R
Contributor guide
Research direction
Start with the R write_parquet entry point and compare its arguments with Python parquet.write_table, focusing on use_byte_stream_split and column_encoding. Determine which requested options can be exposed through the R interface, then verify that they control the resulting Parquet encoding and that existing options remain supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100