apache / apache/arrow

[Python][C++] Support CSV compression in write_dataset

Open
#34,085 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Format Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the usage question you have. Please include as many useful details as possible.

Hi,

I write a dataset to S3 with partitioned on a column as CSV files. But my CSV files should be gzip compressed, I couldn't find anything on docs to do that. Only way it seems, I should write with compressed outputstream however it's not one file.

This is the code I'm using;

```
ds.write_dataset(data, write_path,
format='csv',
partitioning=["p_col"],
partitioning_flavor="hive",
existing_data_behavior='delete_matching')
```

Is there anyway to achieve this to be output gzip files ?

If it's not available, I think it's a good enhancement to consider.

### Component(s)

Format, Python

Contributor guide

Open the contributing guide

Research direction

Start at the Python ds.write_dataset entry point with format='csv' and trace the corresponding C++ dataset-writing path, using the partitioned S3 example as the reproduction case. Examine how compressed output streams are handled, then verify gzip-compressed CSV output for partitioned writes; done means the requested API works and the resulting files are readable.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.