AlexsLemonade / AlexsLemonade/refinebio
Add an experiment discovery layer for SRA to the Surveyor system.
- Dominant language
- Python
- Stars
- 135
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
### Context
The Surveyor system was always supposed to discover experiments from the various sources so that it could get ALL the data. It has made it to this point without this functionality because we haven't been ready to process ALL the data. SurveyJobs were originally intended to do this, but as I implemented them I realized we needed the concept of surveying on an experiment level as well and I never went back updated my abstractions.
### Problem or idea
We need a new job type whose purpose is to discover experiments we want to pull into the system. These jobs will use the various sources' APIs to get a list of experiments, then queue a surveyor job for each one.
This issue is about implementing the discovery jobs for SRA.
### Solution or next step
ENA has a [REST API](https://www.ebi.ac.uk/ena/browse/search-rest) which can be used to query for different SRA data objects. This is my experience the best API of the SRA consortium(DDBJ, ENA, and NCBI SRA). Use the API to retrieve a list of experiments and queue a surveyor job for each one. The first job should query for all experiments from the beginning of time to the present. After that each job should query from the `end_time` of the last job to the present.
### New Issue Checklist
- [x] The title is short and descriptive.
- [x] You have explained the context that led you to write this issue.
- [x] You have reported a problem or idea.
- [x] You have proposed a solution or next step.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the SurveyJob abstractions and existing source integrations described in this issue. Read the ENA REST API documentation and determine how SRA experiments should be discovered and queued for surveying. Done means an initial all-history discovery and subsequent discoveries beginning at the previous job's end_time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100