Imageomics / Imageomics/parquet-cli
Support all compression types for commonly used Parquet creation tools.
- Dominant language
- Dockerfile
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Polars enables compression types [including](https://docs.pola.rs/py-polars/html/reference/api/polars.DataFrame.write_parquet.html):
`{‘lz4’, ‘uncompressed’, ‘snappy’, ‘gzip’, ‘lzo’, ‘brotli’, ‘zstd’}`
Pandas enables compression types [including](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html):
`None, ‘gzip’, ‘brotli’, ‘lz4’, ‘zstd’.`
Pyspark enables compression types [including](https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrameWriter.parquet.html):
`(none, uncompressed, snappy, gzip, lzo, brotli, lz4, and zstd)`
Tested and working:
- uncompressed
- lz4
- snappy
- gzip
- zstd
Tested and not working:
- brotli
Untested:
- lzo
I believe this is determined by the libraries in the base image. Using Ubuntu vs Alpine solved some but not all.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by inspecting the container's base-image configuration and the compression behavior for the listed Parquet creation tools. Reproduce the reported working and failing cases, including brotli and lzo, and compare Ubuntu with Alpine. Done means all requested compression types work through the CLI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- cli, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100