astronomy-commons / astronomy-commons/lsdb

Docs: add recommendations about parquet writing options

Open
#1,402 0 comments 1 reaction 0 assignees View on GitHub
documentation performance
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

For both LSDB and hats-import we should provide some tips on `pyarrow.parquet.write_table` option optimization.

For example, I have learned recently that for list-columns with repeated lists enabling `use_byte_stream_split=[list_column], use_dictionary=[col for col in columns if col != list_column], compression='ZSTD', compression_level=15` leads to incredibly efficient compression. I've tested it on [this file](https://huggingface.co/datasets/MultimodalUniverse/desi/blob/main/edr_sv3/train-00000-of-00007.parquet) and have improved the compression ratio from ×2.5 to ×66 for `spectrum.lambda` column, first chunk.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the documentation for both LSDB and hats-import and identify where parquet-writing guidance belongs. Start by reviewing the pyarrow.parquet.write_table options and the provided compression example, then document applicable recommendations for list-columns and other columns in both projects. Done means contributors can find and apply the tips when writing parquet files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.