apache / apache/arrow

[Python] write_to_dataset(max_row_group_size)

Open
#47,361 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I converted from using `pyarrow.dataset.write_dataset` to `pyarrow.parquet.write_to_dataset` and noticed that this section of code is overwriting my max_row_group_size param to None. It seems it forces the user to pass in chunksize or row_group_size. I think an additional kwargs.pop would be necessary here.

https://github.com/apache/arrow/blob/71a7b5531fd6ef96ce52d04491656031eb023c09/python/pyarrow/parquet/core.py#L2194-L2196

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.