apache / apache/datafusion-comet
Respect Spark compression settings in native Parquet writer
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
https://github.com/apache/datafusion-comet/pull/2812 adds basic support for native Parquet writes, but does not fully support compression codec configs from Spark. Also, lz4 compression level is not configurable.
### Describe the potential solution
Respect Spark configuration options.
Also add a Comet conf for specifying lz4 compression level.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.