apache / apache/datafusion-python
ADD gz decompression in parallel like pigz
Open
enhancement
- Dominant language
- Python
- Stars
- 604
- Forks
- 174
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 4
Description
Hi,
In our case, we would like to read a big CSV file compressed in .gz format.
We would like to use the `read_csv ` function like this:
`ctx.read_csv('myfile.csv.gz',file_extension=".csv.gz", delimiter=';', has_header=True, schema_infer_max_records=0, file_compression_type='gzip')`
However, this decompression is not parallel like pigz (54 seconds) compared to 800 seconds when using the `read_csv` function.
If you could take a look...
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.