astronomy-commons / astronomy-commons/lsdb
Docs: add recommendations about parquet writing options
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
For both LSDB and hats-import we should provide some tips on `pyarrow.parquet.write_table` option optimization.
For example, I have learned recently that for list-columns with repeated lists enabling `use_byte_stream_split=[list_column], use_dictionary=[col for col in columns if col != list_column], compression='ZSTD', compression_level=15` leads to incredibly efficient compression. I've tested it on [this file](https://huggingface.co/datasets/MultimodalUniverse/desi/blob/main/edr_sv3/train-00000-of-00007.parquet) and have improved the compression ratio from ×2.5 to ×66 for `spectrum.lambda` column, first chunk.
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the documentation for both LSDB and hats-import and identify where parquet-writing guidance belongs. Start by reviewing the pyarrow.parquet.write_table options and the provided compression example, then document applicable recommendations for list-columns and other columns in both projects. Done means contributors can find and apply the tips when writing parquet files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100