exp list: does not show new experiments
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Bug Report
Description
When the terminal is killed while dvc exp run is executing, the ref .git/refs/exps/exec/EXEC_BASELINE is not removed.
Then when a git commit is made, git might pack the references to optimize performance.
Now, dvc exp list is stuck with the list of experiments before the commit and will not update when new experiments are run.
This also affects the experiments table in the vscode extension.
Reproduce
git initdvc initdvc stage add -n prepare -d prepare.py python prepare.py- create file prepare.py and write a program that takes some time (e.g.
time.sleep(10)) git add .git commit -m "commit 1"dvc exp run- while running: Kill the terminal (not via ctrl+c but by closing the terminal)
- edit prepare.py (to make dvc exp run execute the pipeline again)
git add .git commit -m "commit 2"git pack-refs --all: when committing, git sometimes does "git pack-refs" for optimization. It can happen right here. To simulate the automatic packing, run git pack-refs --alldvc exp rundvc exp list
Expected
dvc exp list should show the experiment from 13. Instead, it returns nothing.
It only shows the experiment with dvc exp list -A
Environment information
Output of dvc doctor:
$ dvc doctor
DVC version: 2.38.1 (exe)
---------------------------------
Platform: Python 3.10.9 on Windows-10-10.0.19045-SP0
Subprojects:
Supports:
azure (adlfs = 2022.11.2, knack = 0.10.1, azure-identity = 1.12.0),
gdrive (pydrive2 = 1.15.0),
gs (gcsfs = 2022.11.0),
hdfs (fsspec = 2022.11.0, pyarrow = 10.0.1),
http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
oss (ossfs = 2023.1.0),
s3 (s3fs = 2022.11.0, boto3 = 1.24.59),
ssh (sshfs = 2022.6.0),
webdav (webdav4 = 0.9.8),
webdavs (webdav4 = 0.9.8),
webhdfs (fsspec = 2022.11.0)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue using the listed git and DVC commands, especially the packed refs step, then start from the dvc exp list entry point and compare it with dvc exp list -A. Done means a newly run experiment appears in the default listing after refs are packed, and the VS Code experiments table is no longer affected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100