lincc-frameworks / lincc-frameworks/rubin-dash
Investigate using BYTE_STREAM_SPLIT encoding for float columns
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
It was suggested that byte stream split encoding may perform better for float columns without repeated values, we should investigate the impact this would have and if we could use it with the dash output catalogs to improve compression.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the code that writes the dash output catalogs and identifying how float columns are currently encoded. Compare compression and performance for BYTE_STREAM_SPLIT with representative catalogs, then document whether the results justify using it and whether the output remains compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100