Imageomics / Imageomics/parquet-cli

Support all compression types for commonly used Parquet creation tools.

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Dockerfile
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Polars enables compression types [including](https://docs.pola.rs/py-polars/html/reference/api/polars.DataFrame.write_parquet.html):
`{‘lz4’, ‘uncompressed’, ‘snappy’, ‘gzip’, ‘lzo’, ‘brotli’, ‘zstd’}`

Pandas enables compression types [including](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html):
`None, ‘gzip’, ‘brotli’, ‘lz4’, ‘zstd’.`

Pyspark enables compression types [including](https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrameWriter.parquet.html):
`(none, uncompressed, snappy, gzip, lzo, brotli, lz4, and zstd)`

Tested and working:
- uncompressed
- lz4
- snappy
- gzip
- zstd

Tested and not working:
- brotli

Untested:
- lzo

I believe this is determined by the libraries in the base image. Using Ubuntu vs Alpine solved some but not all.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the container's base-image configuration and the compression behavior for the listed Parquet creation tools. Reproduce the reported working and failing cases, including brotli and lzo, and compare Ubuntu with Alpine. Done means all requested compression types work through the CLI.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
cli, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.