Optimize import-url on directories, do not download existing files again
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
The use case is described here:
https://discord.com/channels/485586884165107732/485596304961962003/773882224248225812
https://discord.com/channels/485586884165107732/485596304961962003/776205388654968892
I have a bucket with images that is being updated (new images, etc) regularly:
gs://bucket/dataset/images
I don't want to use dvc add, primarily because I have other projects, websites using this bucket directly.
I still want to use these images in the DVC project, download them, download new, etc (basically do sync operation):
dvc run -n download_images
--outs-persist-no-cache dataset/images
"gsutil -m cp -n -r gs://bucket/dataset/images dataset/"
# -m -> multiprocess
# -n -> not downloading existing files (even though i wouldnt care because i pay the access anyway and the traffic is free) but its faster
This can be done with import-url. This is what import-url was made for:
dvc import-url gs://bucket/dataset/images
The problem is that import-url downloads all the images again on every change in the remote directory.
We should somehow utilize existing knowledge about the local cache to avoid downloading existing images and download only changes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing repeated dvc import-url gs://bucket/dataset/images runs against a changing directory and compare the existing local files with newly added images. Trace the import-url command and its local-cache handling; done means unchanged files are not downloaded again while new or changed remote files are synchronized.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100