AlexsLemonade / AlexsLemonade/refinebio
Compress individual files inside compendia
- Dominant language
- Python
- Stars
- 135
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
### Context
I downloaded the refine.bio homo sapiens compendium to compare metadata labels with other resources, like MetaSRA.
### Problem
The human compendium zip file is very large (~ 48Gb), and unzipping to access the contents resulted in my laptop running out of disk space! I wish there was a way to access the data I need while not causing further disk space issues.
### Solution
In conversation with @cgreene and @jaclyn-taroni , one idea would be to compress the individual files within the zip download before zipping all the compendium files together. They would be readable using command line utilities without having to be expanded. Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing how compendium downloads are assembled and how the individual files are stored inside the final archive. Compare the proposed per-file compression approach with the requirement that command-line utilities can read the contents without expansion; done means users can access individual files without extracting the full compendium.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100