OpenFreeEnergy / OpenFreeEnergy/openfe

Nonspecific rmtree failure when accidentally resuming a job twice

Open
#1,912 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
331
Forks
56
Avg merge
3d 9h
Merged PRs (30d)
13

Description

If you try to quickrun resume the same transformation twice, you can end up in a situation where two parallel attempts at completing the job are happening.

When that happens, one of the jobs can eventually fail with this kind of stack trace:

Traceback (most recent call last):
  File "/home/ialibay/software/mambaforge/install/envs/openfe/bin/openfe", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
  File "/home/ialibay/github/openfe/src/openfecli/utils.py", line 116, in wrapper
    result = function(*args, **kwargs)
  File "/home/ialibay/github/openfe/src/openfecli/commands/quickrun.py", line 160, in quickrun
    dagresult = execute_DAG(
        dag,
    ...<5 lines>...
        n_retries=2,
    )
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/site-packages/gufe/protocols/protocoldag.py", line 526, in execute_DAG
    shutil.rmtree(scratch)
    ~~~~~~~~~~~~~^^^^^^^^^
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/shutil.py", line 763, in rmtree
    _rmtree_safe_fd(stack, onexc)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/shutil.py", line 704, in _rmtree_safe_fd
    onexc(func, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^
  File "/home/ialibay/software/mambaforge/install/envs/openfe/lib/python3.13/shutil.py", line 665, in _rmtree_safe_fd
    orig_st = os.lstat(name, dir_fd=dirfd)
FileNotFoundError: [Errno 2] No such file or directory: PosixPath('results/test/scratch_HybridTopologyMultiStateSimulationUnit-d254ea3d8d0a495bb69ed7c57c219520_attempt_0')

This is probably because the other runner has already deleted the scratch repository. It may make sense to capture that error and provide users with a clearer error.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/openfecli/commands/quickrun.py around line 160, then follow the execute_DAG call shown in the traceback into gufe's protocoldag.py. Reproduce a duplicated quickrun resume if possible and inspect how scratch cleanup reaches shutil.rmtree. Done means the duplicate-resume failure reports a clearer user-facing error instead of the nonspecific FileNotFoundError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.