treeverse / treeverse/dvc

exp list: does not show new experiments

Open
#9,260 6 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: experiments bug p2-medium
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

Description

When the terminal is killed while dvc exp run is executing, the ref .git/refs/exps/exec/EXEC_BASELINE is not removed.
Then when a git commit is made, git might pack the references to optimize performance.
Now, dvc exp list is stuck with the list of experiments before the commit and will not update when new experiments are run.

This also affects the experiments table in the vscode extension.

Reproduce
  1. git init
  2. dvc init
  3. dvc stage add -n prepare -d prepare.py python prepare.py
  4. create file prepare.py and write a program that takes some time (e.g. time.sleep(10))
  5. git add .
  6. git commit -m "commit 1"
  7. dvc exp run
  8. while running: Kill the terminal (not via ctrl+c but by closing the terminal)
  9. edit prepare.py (to make dvc exp run execute the pipeline again)
  10. git add .
  11. git commit -m "commit 2"
  12. git pack-refs --all: when committing, git sometimes does "git pack-refs" for optimization. It can happen right here. To simulate the automatic packing, run git pack-refs --all
  13. dvc exp run
  14. dvc exp list
Expected

dvc exp list should show the experiment from 13. Instead, it returns nothing.
It only shows the experiment with dvc exp list -A

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.38.1 (exe)
---------------------------------
Platform: Python 3.10.9 on Windows-10-10.0.19045-SP0
Subprojects:

Supports:
        azure (adlfs = 2022.11.2, knack = 0.10.1, azure-identity = 1.12.0),
        gdrive (pydrive2 = 1.15.0),
        gs (gcsfs = 2022.11.0),
        hdfs (fsspec = 2022.11.0, pyarrow = 10.0.1),
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        oss (ossfs = 2023.1.0),
        s3 (s3fs = 2022.11.0, boto3 = 1.24.59),
        ssh (sshfs = 2022.6.0),
        webdav (webdav4 = 0.9.8),
        webdavs (webdav4 = 0.9.8),
        webhdfs (fsspec = 2022.11.0)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue using the listed git and DVC commands, especially the packed refs step, then start from the dvc exp list entry point and compare it with dvc exp list -A. Done means a newly run experiment appears in the default listing after refs are packed, and the VS Code experiments table is no longer affected.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.