galaxyproject / galaxyproject/galaxy

Data Libraries: Importing from path copies already existing datasets

Open
#6,589 0 comments 0 reactions 0 assignees View on GitHub
area/libraries feature-request
Dominant language
Python
Stars
1.9k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
169

Description

I have a system path which contains data. Let's call it `/data`. I have a particular experiment which I'm receiving data from in batches. Each time I get new data from this experiment, I put it in `/data/experiment1`. In my galaxy, I have a data library called `"Experiment 1"` and I import data by "Add Datasets > From Path > Link Files Instead Of Copying".

However, once a new batch of Experiment1 data comes in, if I do "Add Datasets > From Path > Link Files Instead Of Copying" from `/data/experiment1` galaxy re-imports previously imported datasets.

E.g.
Say the first file I get as part of Experiment 1 is A.fastq.
Now the data library contains A.fastq.
Say I now get a new file called B.fastq
After an import, now the data library contains [A.fastq, A.fastq, B.fastq].
now I get C.fastq and again import `/data/experiment1`
Now the data library contains [A.fastq, A.fastq, A.fastq, B.fastq, B.fastq, C.fastq]
And on and on...

It should be feasible to examine unix metadata associated with files to check whether datasets have been previously linked in data libraries, right?

Contributor guide

Open the contributing guide

Research direction

The issue does not name files, tests, or an entry point. Start by reproducing the Data Libraries “From Path” import with linked files and trace how existing datasets are detected; done means rerunning the import adds only new files such as B.fastq and C.fastq without duplicating earlier files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.