treeverse / treeverse/dvc

Feature Request: `dvc exp push --queued` to push queued experiments

Open
#10,876 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

I want to run my experiments in my CI/CD pipeline, so I can use remote compute resources and also share the results with my team. My intended workflow is the following:

# Create a branch for the experiments
$ git checkout -b my-experiment

# Queue up some experiments
$ dvc exp run --name my-experiment --queue "param=1,2,3,4"

# Push the queue. This is the FEATURE REQUEST.
$ dvc exp push --queued

# Open a PR
$ git commit -am "Experiment description" --allow-empty
$ git push origin my-experiment
$ gh pr create --fill

Then a CI/CD pipeline will basically run:

# Before this command we check if there are any experiments queued
# ...
# Run all the experiments and show results
$ dvc exp run --run-all
$ dvc exp show

After reading and deciding an experiment to apply, I run locally:

# Pull experiment results and apply the best one
$ dvc exp pull
$ dvc apply best-experiment
$ git commit -am "Apply best parameters"
$ git push origin my-experiment

# After my team reviews the PR
$ gh pr merge

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing dvc exp push command and the queued experiment workflow created by dvc exp run --queue. Define how --queued should transfer queued experiments so the CI commands dvc exp run --run-all and dvc exp show can consume them, then verify the workflow through the documented branch and pull-request sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
ci-cd, cli, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.