[Python][C++] Support CSV compression in write_dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the usage question you have. Please include as many useful details as possible.
Hi,
I write a dataset to S3 with partitioned on a column as CSV files. But my CSV files should be gzip compressed, I couldn't find anything on docs to do that. Only way it seems, I should write with compressed outputstream however it's not one file.
This is the code I'm using;
```
ds.write_dataset(data, write_path,
format='csv',
partitioning=["p_col"],
partitioning_flavor="hive",
existing_data_behavior='delete_matching')
```
Is there anyway to achieve this to be output gzip files ?
If it's not available, I think it's a good enhancement to consider.
### Component(s)
Format, Python
Contributor guide
Research direction
Start at the Python ds.write_dataset entry point with format='csv' and trace the corresponding C++ dataset-writing path, using the partitioned S3 example as the reproduction case. Examine how compressed output streams are handled, then verify gzip-compressed CSV output for partitioned writes; done means the requested API works and the resulting files are readable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100