AlexsLemonade / AlexsLemonade/refinebio

Add an experiment discovery layer for SRA to the Surveyor system.

Open
#226 0 comments 0 reactions 0 assignees View on GitHub
backlog surveyor
Dominant language
Python
Stars
135
Forks
21
PR merge metrics
No merged PRs in 30d

Description

### Context

The Surveyor system was always supposed to discover experiments from the various sources so that it could get ALL the data. It has made it to this point without this functionality because we haven't been ready to process ALL the data. SurveyJobs were originally intended to do this, but as I implemented them I realized we needed the concept of surveying on an experiment level as well and I never went back updated my abstractions.

### Problem or idea

We need a new job type whose purpose is to discover experiments we want to pull into the system. These jobs will use the various sources' APIs to get a list of experiments, then queue a surveyor job for each one.

This issue is about implementing the discovery jobs for SRA.

### Solution or next step

ENA has a [REST API](https://www.ebi.ac.uk/ena/browse/search-rest) which can be used to query for different SRA data objects. This is my experience the best API of the SRA consortium(DDBJ, ENA, and NCBI SRA). Use the API to retrieve a list of experiments and queue a surveyor job for each one. The first job should query for all experiments from the beginning of time to the present. After that each job should query from the `end_time` of the last job to the present.

### New Issue Checklist

- [x] The title is short and descriptive.
- [x] You have explained the context that led you to write this issue.
- [x] You have reported a problem or idea.
- [x] You have proposed a solution or next step.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the SurveyJob abstractions and existing source integrations described in this issue. Read the ENA REST API documentation and determine how SRA experiments should be discovered and queued for surveying. Done means an initial all-history discovery and subsequent discoveries beginning at the previous job's end_time.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
bioinformatics, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.