BUG: Lock Dir

2024-11-09 03:23:10,955 (73b4ed6f99d5:runner): WARNING/MainProcess - resources.py - Lock dir "/var/lib/terra/lock/73b4ed6f99d5/7/gpu" is not empty. Deleting it now for soft-lock support.
Exception in runner on 73b4ed6f99d5
Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
  exec(code, run_globals)
File "/src/dsm/service_runners/crop_and_calibrate.py", line 18, in
  from dsm.tasks.crop_and_calibrate import crop_and_calibrate_task
File "/src/dsm/tasks/init.py", line 16, in
  ResourceManager.register_resource('gpu', GPU_INDICES, WORKERS_PER_GPU)
File "/src/external/terra/terra/executor/resources.py", line 438, in register_resource
  resource = Resource(name, *args, **kwargs)
File "/src/external/terra/terra/executor/resources.py", line 169, in init
  rmtree(self.lock_dir)
File "/usr/local/lib/python3.10/shutil.py", line 725, in rmtree
  _rmtree_safe_fd(fd, path, onerror)
File "/usr/local/lib/python3.10/shutil.py", line 681, in _rmtree_safe_fd
  onerror(os.unlink, fullname, sys.exc_info())
File "/usr/local/lib/python3.10/shutil.py", line 679, in _rmtree_safe_fd
  os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'tmp5dj5n3rv'
Exception ignored in: <function Resource.del at 0x7fc1877f1b40>
Traceback (most recent call last):
File "/src/external/terra/terra/executor/resources.py", line 333, in del
  if self.is_locked:
File "/src/external/terra/terra/executor/resources.py", line 203, in is_locked
  if self._local.lock is None:
AttributeError: 'Resource' object has no attribute '_local'