galaxyproject / galaxyproject/galaxy

Request: Paste/Fetch - allow 1 upload job per URL instead of 1 job for all URLs

Open
#6,473 2 comments 0 reactions 0 assignees View on GitHub
area/UI-UX
Dominant language
Python
Stars
1.9k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
169

Description

Hello,

I was importing some files from SRA with Paste/Fetch, by pasting the URLs into the text box, and noticed that if I pasted all links (12 files) into the URL box (see screenshot below) it seems to run as 1 big job that took > 40 mins:

screen shot 2018-07-08 at 12 28 28 pm

And if I pasted one URL per text box like below I could get the files to upload as separate jobs that then took less than 10 mins to the load the 12 files.

screen shot 2018-07-07 at 7 49 44 pm

I like the faster method but it's a little bit of a pain to have to copy each link into a separate text box in the Paste/Fetch so wondering if there could be an option for pasting all URLs into 1 text box (like in the first screenshot) and then having 1 upload job per URL instead of 1 job for all?

(with the 1 upload job per URL, as in the second screenshot above, I think it's also nice to see the datasets turning green quickly in the history as each file uploads, rather than having all datasets spinning yellow in the history for > 40 mins & not knowing if the upload is working or not, which is what happens with the method in the first screenshot)

Some info:

First screenshot leads to upload of all URLs together:
python '/usr/local/galaxy/galaxy-dist/tools/data_source/upload.py' /usr/local/galaxy/galaxy-dist /data/dnb01/galaxy_db/job_working_directory/004/057/4057529/registry.xml /data/dnb01/galaxy_db/job_working_directory/004/057/4057529/upload_params.json 7050403:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050403_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050403.dat 7050404:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050404_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050404.dat 7050405:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050405_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050405.dat 7050406:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050406_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050406.dat 7050407:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050407_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050407.dat 7050408:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050408_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050408.dat 7050409:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050409_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050409.dat 7050410:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050410_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050410.dat 7050411:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050411_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050411.dat 7050412:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050412_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050412.dat 7050413:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050413_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050413.dat 7050414:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050414_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050414.dat

Files I was using:
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/004/SRR1552444/SRR1552444.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/005/SRR1552445/SRR1552445.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/006/SRR1552446/SRR1552446.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/007/SRR1552447/SRR1552447.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/008/SRR1552448/SRR1552448.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/009/SRR1552449/SRR1552449.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/000/SRR1552450/SRR1552450.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/001/SRR1552451/SRR1552451.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/002/SRR1552452/SRR1552452.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/003/SRR1552453/SRR1552453.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/004/SRR1552454/SRR1552454.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/005/SRR1552455/SRR1552455.fastq.gz

Contributor guide

Open the contributing guide

Research direction

Start with tools/data_source/upload.py and trace how the Paste/Fetch form submits multiple URLs and creates upload jobs. Verify the behavior with a multi-URL submission: each URL should become a separate job, and datasets should update independently in history.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
full-stack
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.