apache / apache/datafusion-python
ADD gz decompression in parallel like pigz
- 主要語言
- Python
- 星號
- 604
- 分支
- 174
- 平均合併
- 2 天 22 小時
- 30 天內合併 PR
- 5
描述
Hi,
In our case, we would like to read a big CSV file compressed in .gz format.
We would like to use the `read_csv ` function like this:
`ctx.read_csv('myfile.csv.gz',file_extension=".csv.gz", delimiter=';', has_header=True, schema_infer_max_records=0, file_compression_type='gzip')`
However, this decompression is not parallel like pigz (54 seconds) compared to 800 seconds when using the `read_csv` function.
If you could take a look...
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
Start at the ctx.read_csv entry point described in the issue and trace how .gz input is decompressed. Determine where parallel gzip decompression would fit and how it should be measured against the reported pigz and read_csv times. Done means the requested gzip CSV read path provides parallel decompression with validated performance.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- data-engineering
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 30/100