treeverse / treeverse/dvc

add support for wildcard/patterns

Open
#4,816 17 comments 16 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature p2-medium
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Sometimes only a subset of files is needed when the user runs import or pull data from a data directory. It is convenient to define a file pattern for an import.

From https://discuss.dvc.org/t/working-with-a-small-subset-of-remote-data/541
Related: https://github.com/iterative/dvc/issues/4705, https://github.com/iterative/dvc/issues/4815

Patterns to implement:

  • simple wildcard dvc pull cats-dogs/data/train/dogs/*.img
  • whole wildcard dvc pull cats-dogs/data/train/{dogs,cats}/???.img
  • globstar/ricursive dvc pull cats-dogs/data/train/**/*.img
  • iterator dvc pull cats-dogs/data/train/dogs/%C.img?counter=1:100
  • date dvc pull users/%Y/%m/%d/users.csv?startdata=2020-09-01,enddate=now,ignoremissing

The first three patterns should use a regular Unix file syntax. While the last two require a special language to define the pattern - we need to find a good examples.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing related issues 4705 and 4815 and the linked discussion about selecting subsets of remote data. Define the syntax and scope for the three Unix-style patterns and the proposed iterator and date patterns; done means the agreed patterns work with dvc pull and import, including the stated examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.