dvc.api.params_show: LockError: Unable to acquire lock - when running multiple processes with `torchrun`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Bug Report
Description
I'm running a standard torchrun to kick off my python script, and the first thing I do is grab the parameters from dvc using a line like this:
dvc_params = dvc.api.params_show(stages=dvc_stage_name)
Of course, that takes a dvc lock under the covers, and apparently that takes too long sometimes, because I am getting this error:
LockError: Unable to acquire lock. Most likely another DVC process is running or
was terminated abruptly. Check the page
<https://dvc.org/doc/user-guide/troubleshooting#lock-issue> for other
possible reasons and to learn how to resolve this.
When you use torchrun, it kicks off as many processes as there are GPUs, so 8 in this case. So I expect that it would take just a little while for each process to run, although frankly less than the default lock timeout of what appears to be 3 seconds, but I don't know what all dvc is doing under the covers when I call that.
Is there a better way for me to grab the parameters somehow without risking a lock timeout? I can't just look at the params file because I am using the ability to override parameters on the command line.
Reproduce
Launch a python script 8 times simultaneously, with each one calling: ``dvc_params = dvc.api.params_show(stages=dvc_stage_name)`
Expected
I could either avoid the lock timeout by specifying I'm okay with a longer timeout, or this would be fast enough that I could call it across 8 processes without getting a timeout error.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dvc.api.params_show(stages=...) entry point and reproduce the issue by launching eight concurrent Python processes through torchrun. Trace the lock acquisition and timeout behavior used by this call. Done means concurrent parameter reads no longer fail with LockError, or the timeout can be configured as requested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100