treeverse / treeverse/dvc

flatten `deps` to facilitate more flexible templating

Open
#7,151 13 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: templating feature request
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

The templating feature currently admits little flexibility for stage keys (e.g. deps) that expect lists. As a result, foreach stages cannot be given a variable number of dependencies. Causing lists to be flattened would make templating more powerful. For example, you could give extra dependencies to certain templated stages:

stages:
  build:
    foreach:
      us:
        deps:
      uk:
        deps:
        - uk_data
    do:
      cmd: myscript ${key}
      deps:
      - myscript
      - ${item.deps}

I have implemented the change at https://github.com/itcarroll/dvc/blob/feature/flattened-deps/dvc/dependency/__init__.py#L52, by (non-recursive) list flattening in dvc.dependency.loads_from. I am sure it's not the right way to do it, and probably has unintended consequences, but it demonstrates the concept (and no tests failed).

I don't know how templating responds to missing keys, but not having to include the empty list for us would be even more better.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dvc/dependency/init.py, especially loads_from and the proposed non-recursive list-flattening logic around the referenced lines. Review how templated foreach stages handle deps, including missing keys, then add focused coverage showing variable dependency lists and confirming existing dependency behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.