treeverse / treeverse/dvc

Nested matrix substitution

Open
#9,948 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

p3-nice-to-have
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Sometimes evaluating a certain metric calculations can take up some time. So we want to cache these results using dvc. For that we consider the following param.yaml file

experiments:
  experiment_a:
    metrics:
      metric_a_1:
         idx: <str>
         kwargs: <dict>
      [...]
  experiment_b:
    metrics:
      metric_b_1:
         idx: <str>
         kwargs: <dict>
      [...]

combined with the dvc.yaml file

stages:
  dispatch_experiment:
    matrix:
      experiment: ${experiments}
      metric: ${experiments.${item.experiment}.metrics}
   cmd: python -m scripts.dispatch_experiment
      --experiment-name ${item.experiment}
      --metric-name ${item.metric}
   outs:
      - ${item.experiment}/${item.metric}.csv

in order to make the looping of matrix notation more flexible.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the matrix notation referenced in issue #5172 and the example param.yaml and dvc.yaml configurations. Trace how the dispatch_experiment entry point uses nested matrix values; done means nested substitutions expand experiment metrics into the intended stages, commands, and outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, yaml
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.