dask / dask/dask

Re-integrate `pyarrow` `write_to_dataset` instead of `arrow._write_partitioned` in `dask`

Open
#9,968 3 comments 1 reaction 0 assignees View on GitHub
hygiene needs attention parquet
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:

Because of https://github.com/apache/arrow/issues/24440 (https://issues.apache.org/jira/browse/ARROW-8244), dask is using its own code to write a table to a partitioned pyarrow dataset. The code lives here:

https://github.com/dask/dask/blob/3124376d68bd3a15d381ce803ca066c7eef4c24f/dask/dataframe/io/parquet/arrow.py#L89-L101

Since this code was [added in March 2020](https://github.com/dask/dask/pull/6023), `pyarrow` came a long way. The original issue was [fixed in pyarrow in April 2020](https://github.com/apache/arrow/commit/ac3bfe47821cb8368f657860f115e88077eaf64d). There's a TODO in Dask code to re-integrate back `write_to_dataset`, we still need to do it. The custom code already required multiple bugfixes, and we can't be sure that it's up to date with `pyarrow`.

- Dask version: 2023.2.0+11.g0890b96b

cc @rjzamora .

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.