Nested matrix substitution
Open
Nobody has claimed this yet.
p3-nice-to-have
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Sometimes evaluating a certain metric calculations can take up some time. So we want to cache these results using dvc. For that we consider the following param.yaml file
experiments:
experiment_a:
metrics:
metric_a_1:
idx: <str>
kwargs: <dict>
[...]
experiment_b:
metrics:
metric_b_1:
idx: <str>
kwargs: <dict>
[...]
combined with the dvc.yaml file
stages:
dispatch_experiment:
matrix:
experiment: ${experiments}
metric: ${experiments.${item.experiment}.metrics}
cmd: python -m scripts.dispatch_experiment
--experiment-name ${item.experiment}
--metric-name ${item.metric}
outs:
- ${item.experiment}/${item.metric}.csv
in order to make the looping of matrix notation more flexible.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the matrix notation referenced in issue #5172 and the example param.yaml and dvc.yaml configurations. Trace how the dispatch_experiment entry point uses nested matrix values; done means nested substitutions expand experiment metrics into the intended stages, commands, and outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, yaml
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100