imicroseq / imicroseq/singularity

Split Data Release Bundle in to multiple gzips

Open
#42 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Considering the growth of data and Network accessibility required for downloading the data release bundles we want to split the data release output file in multiple batches.

May be restrict bundle sizes to 500MB. The compressed bundle size is ~ 2 GB for 350000 Samples (10 GB actual data)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the data release generation entry point identified by the repository and trace how the single compressed bundle is produced. Define how output should be divided into batches of about 500 MB, then verify that a release containing 350,000 samples can be generated and downloaded as multiple gzip files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.