apache / apache/arrow

[R] write_parquet expose similar options as python parquet.write_table ?

Open
#35,102 2 comments 0 reactions 0 assignees View on GitHub
Component: R Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

I've found R write_parquet

```R
write_parquet(
x,
sink,
chunk_size = NULL,
version = "2.4",
compression = default_parquet_compression(),
compression_level = NULL,
use_dictionary = NULL,
write_statistics = NULL,
data_page_size = NULL,
use_deprecated_int96_timestamps = FALSE,
coerce_timestamps = NULL,
allow_truncated_timestamps = FALSE
)
```

missed some options compared to python parquet.write_table

```python
pyarrow.parquet.write_table(table, where, row_group_size=None, version='2.4', use_dictionary=True, compression='snappy', write_statistics=True, use_deprecated_int96_timestamps=None, coerce_timestamps=None, allow_truncated_timestamps=False, data_page_size=None, flavor=None, filesystem=None, compression_level=None, use_byte_stream_split=False, column_encoding=None, data_page_version='1.0', use_compliant_nested_type=False, encryption_properties=None, write_batch_size=None, dictionary_pagesize_limit=None, store_schema=True, **kwargs)
```

maybe R interface should also expose similar options? the most important options to me is
use_byte_stream_split/ column_encoding, since I'm using R to save parquet file, but it can't specify best column encoding for every column and leads bigger file, I have to use pyarrow to convert it again, which is not very ideal.

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Start with the R write_parquet entry point and compare its arguments with Python parquet.write_table, focusing on use_byte_stream_split and column_encoding. Determine which requested options can be exposed through the R interface, then verify that they control the resulting Parquet encoding and that existing options remain supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.