galaxyproject / galaxyproject/galaxy
Data Libraries: Importing from path copies already existing datasets
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 1.2k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 169
Description
I have a system path which contains data. Let's call it `/data`. I have a particular experiment which I'm receiving data from in batches. Each time I get new data from this experiment, I put it in `/data/experiment1`. In my galaxy, I have a data library called `"Experiment 1"` and I import data by "Add Datasets > From Path > Link Files Instead Of Copying".
However, once a new batch of Experiment1 data comes in, if I do "Add Datasets > From Path > Link Files Instead Of Copying" from `/data/experiment1` galaxy re-imports previously imported datasets.
E.g.
Say the first file I get as part of Experiment 1 is A.fastq.
Now the data library contains A.fastq.
Say I now get a new file called B.fastq
After an import, now the data library contains [A.fastq, A.fastq, B.fastq].
now I get C.fastq and again import `/data/experiment1`
Now the data library contains [A.fastq, A.fastq, A.fastq, B.fastq, B.fastq, C.fastq]
And on and on...
It should be feasible to examine unix metadata associated with files to check whether datasets have been previously linked in data libraries, right?
Contributor guide
Research direction
The issue does not name files, tests, or an entry point. Start by reproducing the Data Libraries “From Path” import with linked files and trace how existing datasets are detected; done means rerunning the import adds only new files such as B.fastq and C.fastq without duplicating earlier files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100