AlexsLemonade / AlexsLemonade/refinebio

Download SRA data from NCBI's S3

Open
#1,372 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
135
Forks
21
PR merge metrics
No merged PRs in 30d

Description

### Context

```
If you haven't seen it, there is an API for looking for locations of files from SRA. It includes the ability to get data from the cloud as signed URLs. I've done some small tests and it appears to work reasonably well. There is also fusera that mounts a cloud bucket as a fuse file system. That also seems to work.

https://t.co/rn7OVxVBEe
```

### Problem or idea

We should use these to download data from SRA, this will let us go way faster.

### Solution or next step

We should probably look into using the Fusera thing to see how performance is when we mount the cloud bucket as a fuse file system. It seems like this could be awesome if can cut out the downloader step but less awesome if it bogs salmon down too much.

Contributor guide

No contributing guide indexed for this repository

Research direction

No implementation files or tests are named. Start by evaluating the NCBI SRA location API and Fusera-mounted S3 access, then measure whether cloud-based access improves download performance without bogging down Salmon; done means a decided and validated approach for replacing or retaining the downloader step.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.