AlexsLemonade / AlexsLemonade/refinebio-py

Downloading unmapped reads contained in `salmontools-results.tar.gz` from ~all RNA seq samples on refine.bio

Open
#79 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
3
PR merge metrics
No merged PRs in 30d

Description

I would love to download ~all of the unmapped reads for RNA-seq samples processed by refine.bio, with the following exceptions:
- I would be fine ignoring GTEX samples
- I would like to ignore bacterial and archaeal samples

I trial'd out downloading the data using this notebook:
https://github.com/taylorreiter/2022-refinebio-unmapped/blob/main/notebooks/try_pyrefinebio.ipynb
The meat of the code is reproduced below:
```
processor_ids = [processor.id for processor in pyrb.Processor.search(name="SALMONTOOLS")]
samples = pyrb.Sample.search(technology="RNA-SEQ")

filepaths = []
accessions = []
computed_filenames = []
downloadurls = []

for sample in samples[0:99]:
for result in sample.results:
if result.processor.id in processor_ids:
computed_files = pyrb.ComputedFile.search(result__id=result.id)

for computed_file in computed_files:
if computed_file.is_qc:
filepaths.append(sample.accession_code + "-" + computed_file.filename)
accessions.append(sample.accession_code)
computed_filenames.append(computed_file.filename)
downloadurls.append(computed_file.download_url)
```

Some challenges I encountered:
+ The loop is very slow. Is there a better way to query refine.bio to retrieve download links? I don't think this approach will scale to all RNA seq data in refinebio
+ Is it possible to only get `salmontools-result.tar.gz` and not also `-multiqc*` files?
+ After running through ~27 of iterations, I received `ServerError: The server encountered an issue`. Do you know what might of caused this, if there is a way to avoid this, or the best way to keep the code running if this is going to happen frequently?

Some questions I have:
+ Are the S3 download links persistent? If not, how long do they last for?
+ I was planning on producing the download links, saving them to a spreadsheet, and then automating their download from S3, but I wasn't sure if this is ill-advised.
+ Does this information already exist is a database or json somewhere? I'm curious if it would be more efficient to work with the database directly considering i'm hoping to download so many files.

Any insight you have would be greatly appreciated!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with notebooks/try_pyrefinebio.ipynb and trace the Processor.search, Sample.search, and ComputedFile.search calls used there. Reproduce the slow queries and ServerError, then inspect the client’s API behavior for filtering, pagination, retries, and download-link handling. Done should be a documented, scalable workflow that retrieves only the requested files and answers the link-persistence questions.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.