galaxyproject / galaxyproject/galaxy
Request: Paste/Fetch - allow 1 upload job per URL instead of 1 job for all URLs
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 1.2k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 169
Description
Hello,
I was importing some files from SRA with Paste/Fetch, by pasting the URLs into the text box, and noticed that if I pasted all links (12 files) into the URL box (see screenshot below) it seems to run as 1 big job that took > 40 mins:

And if I pasted one URL per text box like below I could get the files to upload as separate jobs that then took less than 10 mins to the load the 12 files.

I like the faster method but it's a little bit of a pain to have to copy each link into a separate text box in the Paste/Fetch so wondering if there could be an option for pasting all URLs into 1 text box (like in the first screenshot) and then having 1 upload job per URL instead of 1 job for all?
(with the 1 upload job per URL, as in the second screenshot above, I think it's also nice to see the datasets turning green quickly in the history as each file uploads, rather than having all datasets spinning yellow in the history for > 40 mins & not knowing if the upload is working or not, which is what happens with the method in the first screenshot)
Some info:
First screenshot leads to upload of all URLs together:
python '/usr/local/galaxy/galaxy-dist/tools/data_source/upload.py' /usr/local/galaxy/galaxy-dist /data/dnb01/galaxy_db/job_working_directory/004/057/4057529/registry.xml /data/dnb01/galaxy_db/job_working_directory/004/057/4057529/upload_params.json 7050403:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050403_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050403.dat 7050404:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050404_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050404.dat 7050405:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050405_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050405.dat 7050406:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050406_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050406.dat 7050407:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050407_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050407.dat 7050408:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050408_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050408.dat 7050409:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050409_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050409.dat 7050410:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050410_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050410.dat 7050411:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050411_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050411.dat 7050412:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050412_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050412.dat 7050413:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050413_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050413.dat 7050414:/data/dnb01/galaxy_db/job_working_directory/004/057/4057529/dataset_7050414_files:/data/dnb01/galaxy_db/files/007/050/dataset_7050414.dat
Files I was using:
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/004/SRR1552444/SRR1552444.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/005/SRR1552445/SRR1552445.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/006/SRR1552446/SRR1552446.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/007/SRR1552447/SRR1552447.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/008/SRR1552448/SRR1552448.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/009/SRR1552449/SRR1552449.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/000/SRR1552450/SRR1552450.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/001/SRR1552451/SRR1552451.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/002/SRR1552452/SRR1552452.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/003/SRR1552453/SRR1552453.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/004/SRR1552454/SRR1552454.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR155/005/SRR1552455/SRR1552455.fastq.gz
Contributor guide
Research direction
Start with tools/data_source/upload.py and trace how the Paste/Fetch form submits multiple URLs and creates upload jobs. Verify the behavior with a multi-URL submission: each URL should become a separate job, and datasets should update independently in history.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- full-stack
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100