treeverse / treeverse/dvc

dvc.api.params_show: LockError: Unable to acquire lock - when running multiple processes with `torchrun`

Open
#9,723 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: api p2-medium
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

Description

I'm running a standard torchrun to kick off my python script, and the first thing I do is grab the parameters from dvc using a line like this:
dvc_params = dvc.api.params_show(stages=dvc_stage_name)

Of course, that takes a dvc lock under the covers, and apparently that takes too long sometimes, because I am getting this error:

LockError: Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly. Check the page <https://dvc.org/doc/user-guide/troubleshooting#lock-issue> for other possible reasons and to learn how to resolve this.

When you use torchrun, it kicks off as many processes as there are GPUs, so 8 in this case. So I expect that it would take just a little while for each process to run, although frankly less than the default lock timeout of what appears to be 3 seconds, but I don't know what all dvc is doing under the covers when I call that.

Is there a better way for me to grab the parameters somehow without risking a lock timeout? I can't just look at the params file because I am using the ability to override parameters on the command line.

Reproduce

Launch a python script 8 times simultaneously, with each one calling: ``dvc_params = dvc.api.params_show(stages=dvc_stage_name)`

Expected

I could either avoid the lock timeout by specifying I'm okay with a longer timeout, or this would be fast enough that I could call it across 8 processes without getting a timeout error.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the dvc.api.params_show(stages=...) entry point and reproduce the issue by launching eight concurrent Python processes through torchrun. Trace the lock acquisition and timeout behavior used by this call. Done means concurrent parameter reads no longer fail with LockError, or the timeout can be configured as requested.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.