AlexsLemonade / AlexsLemonade/refinebio

Compress individual files inside compendia

Open
#2,489 2 comments 0 reactions 0 assignees View on GitHub
compendia exploration
Dominant language
Python
Stars
135
Forks
21
PR merge metrics
No merged PRs in 30d

Description

### Context

I downloaded the refine.bio homo sapiens compendium to compare metadata labels with other resources, like MetaSRA.

### Problem

The human compendium zip file is very large (~ 48Gb), and unzipping to access the contents resulted in my laptop running out of disk space! I wish there was a way to access the data I need while not causing further disk space issues.

### Solution

In conversation with @cgreene and @jaclyn-taroni , one idea would be to compress the individual files within the zip download before zipping all the compendium files together. They would be readable using command line utilities without having to be expanded. Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing how compendium downloads are assembled and how the individual files are stored inside the final archive. Compare the proposed per-file compression approach with the requirement that command-line utilities can read the contents without expansion; done means users can access individual files without extracting the full compendium.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.