Unpacking list templates outside `cmd`

Open
#8,171 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
yaml
Domain
tooling

Research direction

Start by reproducing the two deps examples in dvc.yaml with the those list from params.yaml, using dvc repro. Read the templating documentation linked in the issue and trace how list interpolation is handled; done means the supported behavior is implemented or clearly documented, with the requested syntax decision reflected in the examples.

Written by the indexing model from the issue text.

Description

A: templating

For one project, I am running a foreach stage, where each step may have a different list of dependencies. I wanted to use templating to fill in the list of dependencies, but it seems for whatever I tried, the syntax was not supported. Below is a simple example that doesn't include the foreach:

# params.yaml
those:
  - script.py
  - script2.py

I first tried

# dvc.yaml
stages:
  hello_world:
    cmd: python script.py
    deps: 
      - ${those}
    outs:
      - hello.txt

Running dvc repro I get

>>> dvc repro
ERROR: failed to parse 'stages.hello_world.deps' in 'dvc.yaml':
Cannot interpolate data of type 'list'

This wouldn't have made that much sense to me anyways since it reads like I'm expanding a list within an entry of another list.

I also tried

stages:
  hello_world:
    cmd: python script.py
    deps: ${those}
    outs:
      - hello.txt

and got

>>> dvc repro
'./dvc.yaml' validation failed.

expected a list, in stages -> hello_world -> deps, line 3, 
column 5
  2   hello_world:                                          
  3 │   cmd: python script.py                               
  4 │   deps: ${those}

This one makes more sense since it reads similar to how the expansion in foreach stages works.

Please let me know

  1. Is what I'm trying to achieve already supported? In that case the templating documentation should be updated, since this behavior is not described anywhere.
  2. If this is not currently supported, can this be supported in the future? The latter syntax would be a natural extension of the syntax for templating in foreach stages.
Dominant language
Python
Stars
15.9k
Forks
1.3k
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from treeverse/dvc

All issues in treeverse/dvc

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.