treeverse / treeverse/dvc

hydra composition: Workflow discrepancies

Open
#8,355 11 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: experiments discussion
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

When using Hydra Composition for configuring DVC experiments, there are a few discrepancies with respect to the "regular" params workflow. This could cause confusion to existing users of dvc params when migrating to Hydra Composition:

  • The latest state is not preserved

Without Hydra Composition, if modifications are done via --set-param and then the experiment is persisted, the next exp run with no arguments will reuse the latest modifications applied.

With Hydra Composition, the next exp run with no arguments will still run the composition and dump to params.yaml, overriding the latest --set-param modifications.

Users would need to manually edit the files in hydra.config_dir and/or the default list values in hydra.config_name in order to reflect the latest modifications via --set-param.

  • Source of configuration needs to be tracked separately

Without Hydra Composition, tracking the params is enough. A change in params file would result in a new experiment.

With Hydra Composition, tracking only params.yaml could result in unexpected behavior, as manual modifications to files in hydra.config_dir would not be detected by DVC.

Users would need to also track hydra.config_dir.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Hydra Composition guide and reproduce the described dvc exp run and --set-param workflows using params.yaml, hydra.config_dir, and hydra.config_name. Trace how the latest parameter state and configuration sources are handled; done means both workflows preserve expected changes and detect relevant configuration updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.