treeverse / treeverse/dvc

Optimize import-url on directories, do not download existing files again

Open
#4,889 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

optimize p2-medium performance
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

The use case is described here:

https://discord.com/channels/485586884165107732/485596304961962003/773882224248225812
https://discord.com/channels/485586884165107732/485596304961962003/776205388654968892


I have a bucket with images that is being updated (new images, etc) regularly:

gs://bucket/dataset/images

I don't want to use dvc add, primarily because I have other projects, websites using this bucket directly.
I still want to use these images in the DVC project, download them, download new, etc (basically do sync operation):

dvc run -n download_images 
              --outs-persist-no-cache dataset/images 
              "gsutil -m cp -n -r gs://bucket/dataset/images dataset/"
# -m -> multiprocess
# -n -> not downloading existing files (even though i wouldnt care because i pay the access anyway and the traffic is free) but its faster

This can be done with import-url. This is what import-url was made for:

dvc import-url gs://bucket/dataset/images

The problem is that import-url downloads all the images again on every change in the remote directory.

We should somehow utilize existing knowledge about the local cache to avoid downloading existing images and download only changes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing repeated dvc import-url gs://bucket/dataset/images runs against a changing directory and compare the existing local files with newly added images. Trace the import-url command and its local-cache handling; done means unchanged files are not downloaded again while new or changed remote files are synchronized.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.